[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 385136282 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002283] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.005000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.005000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.005016] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.006008] pid_max: default: 32768 minimum: 301 [ 0.007115] LSM: Security Framework initializing [ 0.008057] Yama: becoming mindful. [ 0.009025] SELinux: Initializing. [ 0.009748] *** VALIDATE selinux *** [ 0.017579] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021855] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023124] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024140] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025114] *** VALIDATE tmpfs *** [ 0.027396] *** VALIDATE proc *** [ 0.028252] *** VALIDATE cgroup *** [ 0.029006] *** VALIDATE cgroup2 *** [ 0.030243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032055] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034016] Spectre V2 : User space: Vulnerable [ 0.035003] Speculative Store Bypass: Vulnerable [ 0.037508] debug: unmapping init [mem 0xffffffffb6659000-0xffffffffb6660fff] [ 0.040000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040628] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041025] ... version: 2 [ 0.041872] ... bit width: 48 [ 0.042010] ... generic registers: 4 [ 0.042943] ... value mask: 0000ffffffffffff [ 0.043020] ... max period: 00007fffffffffff [ 0.044029] ... fixed-purpose events: 3 [ 0.045009] ... event mask: 000000070000000f [ 0.046259] rcu: Hierarchical SRCU implementation. [ 0.048226] smp: Bringing up secondary CPUs ... [ 0.049470] x86: Booting SMP configuration: [ 0.050025] .... node #0, CPUs: #1 #2 #3 [ 0.053194] smp: Brought up 1 node, 4 CPUs [ 0.054773] smpboot: Max logical packages: 1 [ 0.055008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138506] node 0 deferred pages initialised in 81ms [ 0.141132] devtmpfs: initialized [ 0.142145] x86/mm: Memory block size: 128MB [ 0.144893] gcov: version magic: 0x41383552 [ 0.146189] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147045] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148198] pinctrl core: initialized pinctrl subsystem [ 0.149112] [ 0.149550] ************************************************************* [ 0.150010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151008] ** ** [ 0.152009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153005] ** ** [ 0.154005] ** This means that this kernel is built to expose internal ** [ 0.155005] ** IOMMU data structures, which may compromise security on ** [ 0.156010] ** your system. ** [ 0.157011] ** ** [ 0.158006] ** If you see this message and you are not debugging the ** [ 0.159005] ** kernel, report this immediately to your vendor! ** [ 0.160005] ** ** [ 0.161009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162006] ************************************************************* [ 0.163524] NET: Registered protocol family 16 [ 0.164315] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165030] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166047] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.167327] cpuidle: using governor menu [ 0.168864] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.171538] PCI: Using configuration type 1 for base access [ 0.173108] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181099] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.182037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184113] cryptd: max_cpu_qlen set to 1000 [ 0.188291] ACPI: Added _OSI(Module Device) [ 0.189015] ACPI: Added _OSI(Processor Device) [ 0.191007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.192007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.196129] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.202456] ACPI: Interpreter enabled [ 0.204057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205010] ACPI: Using IOAPIC for interrupt routing [ 0.206115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.209429] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.218297] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.220045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.221015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.223087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.227489] acpiphp: Slot [2] registered [ 0.229100] acpiphp: Slot [5] registered [ 0.230140] acpiphp: Slot [6] registered [ 0.232107] acpiphp: Slot [3] registered [ 0.233088] acpiphp: Slot [4] registered [ 0.235075] acpiphp: Slot [7] registered [ 0.236074] acpiphp: Slot [8] registered [ 0.237077] acpiphp: Slot [9] registered [ 0.239106] acpiphp: Slot [10] registered [ 0.240098] acpiphp: Slot [11] registered [ 0.242093] acpiphp: Slot [12] registered [ 0.243071] acpiphp: Slot [13] registered [ 0.244066] acpiphp: Slot [14] registered [ 0.246855] acpiphp: Slot [15] registered [ 0.251370] acpiphp: Slot [16] registered [ 0.252065] acpiphp: Slot [17] registered [ 0.254081] acpiphp: Slot [18] registered [ 0.255050] acpiphp: Slot [19] registered [ 0.256037] acpiphp: Slot [20] registered [ 0.257064] acpiphp: Slot [21] registered [ 0.258112] acpiphp: Slot [22] registered [ 0.260088] acpiphp: Slot [23] registered [ 0.261082] acpiphp: Slot [24] registered [ 0.262064] acpiphp: Slot [25] registered [ 0.264075] acpiphp: Slot [26] registered [ 0.265036] acpiphp: Slot [27] registered [ 0.265934] acpiphp: Slot [28] registered [ 0.267112] acpiphp: Slot [29] registered [ 0.269142] acpiphp: Slot [30] registered [ 0.271120] acpiphp: Slot [31] registered [ 0.272043] PCI host bridge to bus 0000:00 [ 0.273017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.274015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.276016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.278019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.280019] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.282033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.284191] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.286012] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.288860] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.293429] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.297044] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.299015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.301023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.303012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.305522] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.306606] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.308022] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.310696] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.313011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.319018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.323020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.329307] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.342028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.354020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.369023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.382000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.388023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.392011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.404015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.416000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.418307] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.419651] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.422271] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.423260] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.428165] iommu: Default domain type: Passthrough [ 0.429849] SCSI subsystem initialized [ 0.431088] ACPI: bus type USB registered [ 0.432093] usbcore: registered new interface driver usbfs [ 0.434072] usbcore: registered new interface driver hub [ 0.435069] usbcore: registered new device driver usb [ 0.436239] pps_core: LinuxPPS API ver. 1 registered [ 0.438009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.440056] PTP clock support registered [ 0.442099] EDAC MC: Ver: 3.0.0 [ 0.443456] PCI: Using ACPI for IRQ routing [ 0.444631] NetLabel: Initializing [ 0.446006] NetLabel: domain hash size = 128 [ 0.447005] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.449104] NetLabel: unlabeled traffic allowed by default [ 0.450145] vgaarb: loaded [ 0.452007] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.453050] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.460175] clocksource: Switched to clocksource kvm-clock [ 0.556607] VFS: Disk quotas dquot_6.6.0 [ 0.557601] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.559082] *** VALIDATE ramfs *** [ 0.560368] *** VALIDATE hugetlbfs *** [ 0.561780] pnp: PnP ACPI init [ 0.564355] pnp: PnP ACPI: found 6 devices [ 0.581445] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.584130] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.585872] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.587472] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.589303] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.591086] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.593259] NET: Registered protocol family 2 [ 0.595393] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.600179] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.603728] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.610809] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.617163] TCP: Hash tables configured (established 65536 bind 65536) [ 0.621989] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.626957] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.629719] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.633751] NET: Registered protocol family 1 [ 0.643394] RPC: Registered named UNIX socket transport module. [ 0.646292] RPC: Registered udp transport module. [ 0.648896] RPC: Registered tcp transport module. [ 0.650838] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.654050] NET: Registered protocol family 44 [ 0.655764] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.658431] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.661195] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.663237] PCI: CLS 0 bytes, default 64 [ 0.664815] Unpacking initramfs... [ 2.129965] debug: unmapping init [mem 0xffff95437cc64000-0xffff95437ffcffff] [ 2.133844] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.136142] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.139059] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.667229] Initialise system trusted keyrings [ 2.668934] Key type blacklist registered [ 2.670709] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.679834] zbud: loaded [ 2.682703] *** VALIDATE nfs *** [ 2.683922] *** VALIDATE nfs4 *** [ 2.685204] pstore: using deflate compression [ 2.688705] Platform Keyring initialized [ 2.809350] NET: Registered protocol family 38 [ 2.811257] Key type asymmetric registered [ 2.813622] Asymmetric key parser 'x509' registered [ 2.815988] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.819984] io scheduler mq-deadline registered [ 2.821837] io scheduler kyber registered [ 2.825271] io scheduler bfq registered [ 2.829188] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.833790] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.836861] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.840200] ACPI: Power Button [PWRF] [ 2.847680] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.859253] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.870754] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.902935] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.933241] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.937761] Non-volatile memory driver v1.3 [ 2.939245] Linux agpgart interface v0.103 [ 2.981564] virtio_blk virtio1: [vda] 145976 512-byte logical blocks (74.7 MB/71.3 MiB) [ 2.984163] vda: detected capacity change from 0 to 74739712 [ 3.003405] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.007035] vdb: detected capacity change from 0 to 1073741824 [ 3.017356] libphy: Fixed MDIO Bus: probed [ 3.023144] usbcore: registered new interface driver usbserial_generic [ 3.026248] usbserial: USB Serial support registered for generic [ 3.027689] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.030461] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.031898] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.033532] mousedev: PS/2 mouse device common for all mice [ 3.036343] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.038255] rtc_cmos 00:05: RTC can wake from S4 [ 3.042706] rtc_cmos 00:05: registered as rtc0 [ 3.043178] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.044551] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.049251] intel_pstate: CPU model not supported [ 3.052496] hid: raw HID events driver (C) Jiri Kosina [ 3.056301] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.056346] usbcore: registered new interface driver usbhid [ 3.067552] usbhid: USB HID core driver [ 3.070052] drop_monitor: Initializing network drop monitor service [ 3.072068] Initializing XFRM netlink socket [ 3.074690] NET: Registered protocol family 10 [ 3.079270] Segment Routing with IPv6 [ 3.081225] NET: Registered protocol family 17 [ 3.083183] mpls_gso: MPLS GSO support [ 3.088552] RAS: Correctable Errors collector initialized. [ 3.091368] AVX version of gcm_enc/dec engaged. [ 3.093273] AES CTR mode by8 optimization enabled [ 3.180583] sched_clock: Marking stable (3180559428, 0)->(3969789048, -789229620) [ 3.183877] registered taskstats version 1 [ 3.185423] Loading compiled-in X.509 certificates [ 3.187337] zswap: loaded using pool lzo/zbud [ 3.212355] Key type big_key registered [ 3.223897] Key type encrypted registered [ 3.225744] ima: No TPM chip found, activating TPM-bypass! [ 3.227484] ima: Allocated hash algorithm: sha1 [ 3.229332] ima: No architecture policies found [ 3.231351] evm: Initialising EVM extended attributes: [ 3.233236] evm: security.selinux [ 3.234327] evm: security.ima [ 3.235469] evm: security.capability [ 3.236798] evm: HMAC attrs: 0x1 [ 3.239315] rtc_cmos 00:05: setting system clock to 2026-08-11 20:19:18 UTC (1786479558) [ 3.248580] debug: unmapping init [mem 0xffffffffb7603000-0xffffffffb77fffff] [ 3.251491] debug: unmapping init [mem 0xffffffffb6382000-0xffffffffb6658fff] [ 3.263078] Write protecting the kernel read-only data: 28672k [ 3.271083] debug: unmapping init [mem 0xffffffffb4a03000-0xffffffffb4bfffff] [ 3.274013] debug: unmapping init [mem 0xffffffffb5314000-0xffffffffb53fffff] [ 3.309775] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.316198] systemd[1]: Detected virtualization kvm. [ 3.317779] systemd[1]: Detected architecture x86-64. [ 3.319476] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.350221] systemd[1]: No hostname configured. [ 3.352441] systemd[1]: Set hostname to . [ 3.354061] random: systemd: uninitialized urandom read (16 bytes read) [ 3.355790] systemd[1]: Initializing machine ID from random generator. [ 3.453713] random: ln: uninitialized urandom read (6 bytes read) [ 3.563820] hrtimer: interrupt took 14473238 ns [ 3.615276] random: systemd: uninitialized urandom read (16 bytes read) [ 3.617805] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.622588] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.627442] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Swap. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.805420] device-mapper: uevent: version 1.0.3 [ 4.807908] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [[ 5.342428] random: fast init done  OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.147357] virtio_net virtio0 ens2: renamed from eth0 [ 6.337766] scsi host0: ata_piix [ 6.411173] scsi host1: ata_piix [ 6.412493] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.414736] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.742932] random: crng init done [ 10.744244] random: 7 urandom warning(s) missed due to ratelimiting [ 12.750258] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 14.229711] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.118595] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.545841] SELinux: Disabled at runtime. [ 16.624580] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.635811] systemd[1]: Detected virtualization kvm. [ 16.637587] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 17.273507] systemd[1]: initrd-switch-root.service: Succeeded. [ 17.277242] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 17.283542] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 17.286757] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 17.290993] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 17.303225] systemd[1]: Starting Journal Service... Starting Journal Service... [ 17.307757] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ 17.630778] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 18.301855] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 19.010267] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 19.125391] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 19.256857] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.282155] EDAC sbridge: Ver: 1.1.2 [ 21.729779] Key type dns_resolver registered [ 22.118585] NFS: Registering the id_resolver key type [ 22.120535] Key type id_resolver registered [ 22.122028] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg159-client login: [ 55.458872] libcfs: loading out-of-tree module taints kernel. [ 55.528184] Key type ._llcrypt registered [ 55.529315] Key type .llcrypt registered [ 55.799547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 55.809029] alg: No test for adler32 (adler32-zlib) [ 56.802046] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 57.142772] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 58.759133] Key type lgssc registered [ 59.331065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 108.854068] Lustre: Mounted lustre-client - version 2.17.56_51_g4226bd4 [ 111.386263] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 117.509383] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing check_logdir /tmp/testlogs/ [ 119.194341] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing yml_node [ 120.749368] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 121.672662] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 122.591228] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 123.154191] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Aug 11 16:21:17 EDT 2026 [ 128.226734] Lustre: Unmounted lustre-client [ 153.591192] Key type lgssc unregistered [ 153.728887] LNet: 8406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 153.732623] LNetError: 8406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 153.746647] LNet: Removed LNI 192.168.201.59@tcp [ 154.043341] Key type .llcrypt unregistered [ 154.045203] Key type ._llcrypt unregistered [ 164.001208] Key type ._llcrypt registered [ 164.002435] Key type .llcrypt registered [ 164.077529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 164.086592] alg: No test for adler32 (adler32-zlib) [ 164.889575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 164.918900] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 164.928830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 166.487929] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 166.534983] LNet: 9287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 166.537852] LNetError: 9287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 166.546338] LNet: Removed LNI 192.168.201.59@tcp [ 166.825520] Key type .llcrypt unregistered [ 166.827453] Key type ._llcrypt unregistered [ 169.247269] Key type ._llcrypt registered [ 169.248896] Key type .llcrypt registered [ 169.820812] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:22:04 (1786479724) [ 169.987922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 169.995037] alg: No test for adler32 (adler32-zlib) [ 170.803269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.832972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 170.842757] LNet: 9704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 170.850794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 170.872857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 170.880587] LNet: 9711:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 174.669719] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:22:08 (1786479728) [ 174.732709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 174.762180] LNet: 10281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 174.782387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 174.845194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 181.757212] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:22:15 (1786479735) [ 181.960087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 182.002232] LNet: 10872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 182.036271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 182.112966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 187.882604] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:22:22 (1786479742) [ 188.106917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 188.143184] LNet: 11459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 188.180106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 188.381165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 195.965463] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:22:30 (1786479750) [ 196.460579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.472934] LNet: 12043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 196.488543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.530386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 196.742957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 202.872652] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:22:36 (1786479756) [ 202.952708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 203.002527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 203.112669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 203.383956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 203.786640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 211.543422] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:22:45 (1786479765) [ 211.626478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.651283] LNet: 13223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 211.659727] LNet: 13223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 211.717808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.879798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 220.543180] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:22:54 (1786479774) [ 220.736579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.803918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.960700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 221.026759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 221.815949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 221.875662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 222.666799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 222.709810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 223.354225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 223.390431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 224.020362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 224.069354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 224.796612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 224.843000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 225.388763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 225.449102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 226.074275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 226.098447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 226.681767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 226.713332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 227.005653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 227.032286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 227.318254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 227.372692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 227.669837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 227.694980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 227.972364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 227.995546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 234.970621] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:23:08 (1786479788) [ 235.037907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.052253] LNet: 14490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 235.057727] LNet: 14490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 235.071469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.117805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 235.165441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 241.867940] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:23:15 (1786479795) [ 242.037432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 242.081446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.148458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 242.183620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 248.120247] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:23:22 (1786479802) [ 248.477285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 248.503891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 248.570524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 248.595244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 254.317628] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:23:28 (1786479808) [ 254.443131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.469841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.536730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 254.568611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 259.961241] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:23:33 (1786479813) [ 260.009844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.043559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.132826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 260.161476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 264.844282] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:23:38 (1786479818) [ 264.923230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 264.966560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.039641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 265.071097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 270.067410] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:23:44 (1786479824) [ 270.307175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.322116] LNet: 18015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 270.325675] LNet: 18015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 270.340752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.399712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 270.689581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 275.519180] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:23:49 (1786479829) [ 275.720262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.749500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.895802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 276.046173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 279.851628] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:23:54 (1786479834) [ 279.978920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.010515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.090701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 280.196638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 284.123226] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:23:58 (1786479838) [ 284.262911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 284.284814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.576690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 284.702112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 288.589687] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:24:02 (1786479842) [ 288.705963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.737451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.784210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 288.815669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 292.988404] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:24:07 (1786479847) [ 293.208579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.241924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.310958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 293.355845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 298.386642] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:24:12 (1786479852) [ 298.493606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.524704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.578974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 303.384694] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:24:17 (1786479857) [ 303.475154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.506940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.587431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 308.204320] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:24:22 (1786479862) [ 308.277064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.310247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.383955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 308.696738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 312.783555] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:24:27 (1786479867) [ 312.885781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 312.917391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 312.968493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 312.992522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 316.779421] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:24:31 (1786479871) [ 316.967261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.986438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.031542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 317.057232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 321.447686] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:24:35 (1786479875) [ 321.505140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.534538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.587745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 321.614360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 326.120876] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:24:40 (1786479880) [ 326.246371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.267566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.323911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 326.349123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 330.107122] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:24:44 (1786479884) [ 330.352915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.376411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.412757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 330.451383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 330.465307] LNetError: 25811:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 330.493267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 330.508860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 334.171182] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:24:48 (1786479888) [ 334.262054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.311716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.369191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MpY9ivYHO9 [ 334.408232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MpY9ivYHO9 [ 334.431837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MpY9ivYHO9 [ 338.531949] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:24:52 (1786479892) [ 338.797943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.817285] LNet: 26983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.820912] LNet: 26983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 338.849639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.890509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 338.917400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 338.956667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 342.850472] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 16:24:57 (1786479897) [ 343.041934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.081275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.137648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 343.164526] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 344.199531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 344.810471] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 346.527558] Key type lgssc registered [ 347.134946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 349.037519] LNetError: 27573:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.59@tcp99 has no available nets [ 350.348231] Key type lgssc unregistered [ 350.594721] LNetError: 28683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 350.606765] LNet: Removed LNI 192.168.201.59@tcp [ 350.988295] Key type .llcrypt unregistered [ 351.006107] Key type ._llcrypt unregistered [ 354.588689] Key type ._llcrypt registered [ 354.590178] Key type .llcrypt registered [ 355.205526] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:25:09 (1786479909) [ 355.466961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 355.478618] alg: No test for adler32 (adler32-zlib) [ 356.314391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.354630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 357.643233] LNet: Message forwarding will be enabled in 2s [ 360.735154] LNet: Message forwarding enabled [ 360.911200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 360.940769] LNet: Message forwarding will stop in 2s to 6s [ 365.235526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 365.528525] LNet: Message forwarding disabled [ 366.409363] LNet: Message forwarding will be enabled in 2s [ 369.632565] LNet: Message forwarding enabled [ 370.919039] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:25:25 (1786479925) [ 370.983472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.001572] LNet: 30084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.783968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.831832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 375.909647] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:25:30 (1786479930) [ 375.984352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.025522] LNet: 30666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.043245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.119685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 376.157840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 376.178516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 376.202640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 376.226619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 376.250291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 376.268877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 376.290749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 376.310796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 376.333726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 376.354751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 376.376875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 376.405439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 376.431960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 376.449182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 376.477838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 376.498885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 376.526802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 376.548565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 376.564488] LNetError: 30713:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 380.642404] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:25:35 (1786479935) [ 380.825965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.842269] LNet: 31284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.854864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.918760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 384.955428] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:25:39 (1786479939) [ 385.922340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg159-server [ 387.364306] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:25:41 (1786479941) [ 387.533426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.559995] LNet: 32214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.594860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.656649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 387.673495] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 388.461486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[60-64]@tcp [ 388.484810] LNet: 32348:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 388.526107] LNetError: 32218:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.60@tcp is being used as a gateway but routing feature is not turned on [ 388.540502] LNetError: 32218:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.60@tcp has gone from up to down [ 389.549533] LNetError: 32218:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.60@tcp is being used as a gateway but routing feature is not turned on [ 389.557362] LNetError: 32218:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 390.572517] LNetError: 32218:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.60@tcp is being used as a gateway but routing feature is not turned on [ 391.584515] LNetError: 32222:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.61@tcp has gone from up to down [ 392.623307] LNetError: 32218:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.60@tcp is being used as a gateway but routing feature is not turned on [ 392.634828] LNetError: 32218:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 392.871902] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:25:46 (1786479946) [ 394.323149] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg159-server [ 395.463571] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:25:49 (1786479949) [ 395.591733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.609308] LNet: 33283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.613476] LNetError: 33283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 395.625598] LNet: Removed LNI 192.168.201.59@tcp [ 395.639986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.695549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 395.711441] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 396.099386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[60-65/2]@tcp [ 396.112745] LNet: 33400:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 396.118395] LNet: 33400:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 396.130686] LNetError: 33287:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.60@tcp has gone from up to down [ 396.136038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[60-65/2]@tcp [ 396.139032] LNetError: 33287:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 400.410491] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:25:54 (1786479954) [ 400.472550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.490605] LNetError: 33985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 400.520455] LNet: Removed LNI 192.168.201.59@tcp [ 400.552330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.613580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 400.645174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.690723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.747775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 400.786930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.812918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.852226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 400.914777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.953582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.990480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.039766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.063520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.102677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 401.152817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.211298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.290674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.344794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.381667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.439612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 401.507760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.550409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.592886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.639383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.695339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.747363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 401.789764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.809747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.847388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.896608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.937217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.985328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 402.020457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.050751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.108263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 406.583547] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:26:00 (1786479960) [ 407.846562] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg159-server [ 409.342445] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:26:03 (1786479963) [ 411.361505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg159-server [ 413.184184] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:26:06 (1786479966) [ 413.570858] LNet: 35392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.575080] LNet: 35392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 414.596350] Key type .llcrypt unregistered [ 414.597526] Key type ._llcrypt unregistered [ 422.071773] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 422.254205] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 422.262885] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 422.267800] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 424.885876] Key type ._llcrypt registered [ 424.891033] Key type .llcrypt registered [ 425.177256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 425.190921] alg: No test for adler32 (adler32-zlib) [ 426.037192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.072479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 426.088155] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 426.511644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 426.525978] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 427.942837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 427.957481] LNet: Removed LNI 10.1.2.3@tcp [ 428.064408] LNet: 36120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.067867] LNetError: 36120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 428.081263] LNet: Removed LNI 192.168.201.59@tcp [ 428.640645] Key type .llcrypt unregistered [ 428.642109] Key type ._llcrypt unregistered [ 444.294589] Key type ._llcrypt registered [ 444.297857] Key type .llcrypt registered [ 447.496396] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:26:38 (1786479998) [ 449.054349] Key type .llcrypt unregistered [ 449.058555] Key type ._llcrypt unregistered [ 463.061117] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 463.647406] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 463.659750] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 463.680889] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 466.965176] Key type ._llcrypt registered [ 466.970423] Key type .llcrypt registered [ 467.299289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 467.400328] alg: No test for adler32 (adler32-zlib) [ 468.586715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.715118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 468.824497] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 470.898863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 470.985088] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 478.667163] LNet: 37621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 478.669990] LNetError: 37621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 478.672802] LNet: Removed LNI 10.1.2.3@tcp [ 478.682138] LNet: Removed LNI 192.168.201.59@tcp [ 479.827818] Key type .llcrypt unregistered [ 479.830092] Key type ._llcrypt unregistered [ 498.003125] Key type ._llcrypt registered [ 498.005706] Key type .llcrypt registered [ 499.463600] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:27:33 (1786480053) [ 501.119469] Key type .llcrypt unregistered [ 501.120922] Key type ._llcrypt unregistered [ 510.807600] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 510.974315] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 510.983457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 510.999718] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 514.019145] Key type ._llcrypt registered [ 514.022625] Key type .llcrypt registered [ 514.384632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 514.412836] alg: No test for adler32 (adler32-zlib) [ 515.491280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.614394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 515.675314] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 518.512498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 518.540600] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 521.151959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 521.191615] LNet: Removed LNI 10.1.2.3@tcp [ 521.221433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 521.280878] LNet: Removed LNI 10.1.2.31@tcp [ 521.316445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 521.371442] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 522.286171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 522.313242] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 523.747927] LNet: 39552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 523.754400] LNetError: 39552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 523.761052] LNet: Removed LNI 10.1.2.31@tcp [ 523.782814] LNet: Removed LNI 10.1.2.3@tcp [ 524.603453] Key type .llcrypt unregistered [ 524.607893] Key type ._llcrypt unregistered [ 542.466143] Key type ._llcrypt registered [ 542.468953] Key type .llcrypt registered [ 544.461736] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:28:17 (1786480097) [ 545.151203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 545.167985] alg: No test for adler32 (adler32-zlib) [ 546.061631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.097653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 546.118480] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 546.815909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.837178] LNet: 40511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.842717] LNetError: 40511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.857856] LNet: Removed LNI 192.168.201.59@tcp [ 546.877634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.930948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.59@tcp [ 546.950859] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 548.217414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.59@tcp [ 548.239918] LNet: Removed LNI 192.168.201.59@tcp [ 548.458453] LNet: 40646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.394031] Key type .llcrypt unregistered [ 549.415696] Key type ._llcrypt unregistered [ 565.093299] Key type ._llcrypt registered [ 565.100204] Key type .llcrypt registered [ 567.367717] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:28:39 (1786480119) [ 569.631683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg159-server [ 571.959788] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:28:45 (1786480125) [ 572.707317] Key type .llcrypt unregistered [ 572.710672] Key type ._llcrypt unregistered [ 579.307524] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 579.490984] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 579.509668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 579.519533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 581.246592] Key type ._llcrypt registered [ 581.250718] Key type .llcrypt registered [ 581.433512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 581.449248] alg: No test for adler32 (adler32-zlib) [ 582.434655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.536787] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 583.517511] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 585.571388] LNet: 42479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.586838] LNetError: 42479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 585.594854] LNet: Removed LNI 192.168.201.59@tcp1 [ 585.628356] LNet: Removed LNI 10.1.2.3@tcp2 [ 586.462839] Key type .llcrypt unregistered [ 586.466838] Key type ._llcrypt unregistered [ 601.580120] Key type ._llcrypt registered [ 601.581412] Key type .llcrypt registered [ 603.010120] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:29:16 (1786480156) [ 603.441134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 603.465826] alg: No test for adler32 (adler32-zlib) [ 604.517064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.577692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 604.606037] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 605.954481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.998813] LNet: 43453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.025124] LNetError: 43453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.036734] LNet: Removed LNI 192.168.201.59@tcp [ 606.074889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.166592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 606.209064] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 614.269451] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:29:27 (1786480167) [ 614.367512] LNet: 44035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.376580] LNetError: 44035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.406750] LNet: Removed LNI 192.168.201.59@tcp [ 615.069528] Key type .llcrypt unregistered [ 615.072263] Key type ._llcrypt unregistered [ 623.590365] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 623.858127] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 623.863472] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 623.867820] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 626.700291] Key type ._llcrypt registered [ 626.702599] Key type .llcrypt registered [ 626.900393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 626.916200] alg: No test for adler32 (adler32-zlib) [ 627.920556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.008921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 628.035575] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 628.939312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 628.964228] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 631.917631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.948912] LNet: 44792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.976164] LNetError: 44792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 631.982855] LNet: Removed LNI 10.1.2.3@tcp [ 632.014092] LNet: Removed LNI 192.168.201.59@tcp [ 632.047879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.121325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 632.152498] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 632.159096] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 632.733293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.745272] LNet: 44819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.748712] LNetError: 44819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 632.752623] LNetError: 44819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 632.757783] LNet: Removed LNI 10.1.2.3@tcp [ 632.765824] LNet: Removed LNI 192.168.201.59@tcp [ 632.779766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.832944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 632.858775] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 632.862757] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 633.474691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 633.503364] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 634.553964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.573380] LNet: 44981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.579909] LNetError: 44981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 634.586356] LNetError: 44981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 634.593391] LNet: Removed LNI 10.1.2.3@tcp [ 634.596602] LNet: Removed LNI 192.168.201.59@tcp [ 634.605568] LNet: Removed LNI 192.168.201.59@tcp2 [ 634.624461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.699953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 634.727376] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 634.734328] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 634.740651] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 642.920153] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:29:56 (1786480196) [ 643.327980] LNet: 45580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.333642] LNetError: 45580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 643.341682] LNetError: 45580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 643.352258] LNet: Removed LNI 10.1.2.3@tcp [ 643.355806] LNet: Removed LNI 192.168.201.59@tcp [ 643.375429] LNet: Removed LNI 192.168.201.59@tcp2 [ 644.241534] Key type .llcrypt unregistered [ 644.245780] Key type ._llcrypt unregistered [ 651.723343] Key type ._llcrypt registered [ 651.727531] Key type .llcrypt registered [ 652.031645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 652.054728] alg: No test for adler32 (adler32-zlib) [ 653.042098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.128390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 653.168217] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 654.095694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.130395] LNet: 45978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.146795] LNetError: 45978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.173163] LNet: Removed LNI 192.168.201.59@tcp [ 654.214761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.326982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 654.393090] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 655.074308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.128171] LNet: 46008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.158905] LNetError: 46008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.190440] LNet: Removed LNI 192.168.201.59@tcp [ 655.223261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.300136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 655.341730] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 655.680321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.722336] LNet: Removed LNI 192.168.201.59@tcp [ 655.767413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.827983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 655.843592] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 656.285063] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 656.407125] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 656.417899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 656.428650] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 658.926436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.958568] LNet: 46114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.968973] LNet: 46114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 658.980438] LNetError: 46114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.995762] LNetError: 46114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 659.015412] LNet: Removed LNI 192.168.201.59@tcp [ 659.049529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.147834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 659.202508] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 667.933086] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:30:21 (1786480221) [ 668.066363] LNet: 46703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.078293] LNetError: 46703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.105883] LNet: Removed LNI 192.168.201.59@tcp [ 668.896865] Key type .llcrypt unregistered [ 668.904321] Key type ._llcrypt unregistered [ 675.921091] Key type ._llcrypt registered [ 675.922898] Key type .llcrypt registered [ 676.090469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 676.113399] alg: No test for adler32 (adler32-zlib) [ 677.079745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.131159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 677.153171] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 678.173615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.240432] LNet: 47115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.255368] LNetError: 47115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.272925] LNet: Removed LNI 192.168.201.59@tcp [ 678.307762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.442048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 678.500612] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 685.502781] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:30:39 (1786480239) [ 685.918571] LNet: 47707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.925210] LNetError: 47707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.938384] LNet: Removed LNI 192.168.201.59@tcp [ 686.777758] Key type .llcrypt unregistered [ 686.781330] Key type ._llcrypt unregistered [ 693.075521] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 693.328428] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 693.339746] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 693.345757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 696.143626] Key type ._llcrypt registered [ 696.146842] Key type .llcrypt registered [ 696.445314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 696.471345] alg: No test for adler32 (adler32-zlib) [ 697.399734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.483985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.514769] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 698.714070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.748238] LNet: 48160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.756173] LNetError: 48160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.770852] LNet: Removed LNI 192.168.201.59@tcp [ 698.808877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.923974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 698.963296] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 706.920867] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:31:00 (1786480260) [ 707.077912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.116269] LNet: 48755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.123803] LNetError: 48755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.143605] LNet: Removed LNI 192.168.201.59@tcp [ 707.183927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.295342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 714.890983] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:31:08 (1786480268) [ 715.214354] LNet: 49342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.974577] Key type .llcrypt unregistered [ 715.978372] Key type ._llcrypt unregistered [ 723.531268] Key type ._llcrypt registered [ 723.539985] Key type .llcrypt registered [ 723.890073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 723.920095] alg: No test for adler32 (adler32-zlib) [ 724.889829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.010623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.026322] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 725.822433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.846251] LNet: 49743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.851308] LNetError: 49743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.864931] LNet: Removed LNI 192.168.201.59@tcp [ 725.884843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.950139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 725.975629] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 733.177443] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:31:27 (1786480287) [ 733.409428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 733.475851] LNet: 50334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.503176] LNetError: 50334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.563187] LNet: Removed LNI 192.168.201.59@tcp [ 733.589868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.652771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 739.614414] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:31:33 (1786480293) [ 741.623946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 743.567744] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:31:37 (1786480297) [ 745.332130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 747.187804] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:31:41 (1786480301) [ 748.749388] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 750.904524] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:31:44 (1786480304) [ 752.562951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 754.491225] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:31:48 (1786480308) [ 756.149253] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 757.848318] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:31:51 (1786480311) [ 758.015426] LNet: 52388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.616639] Key type .llcrypt unregistered [ 758.623305] Key type ._llcrypt unregistered [ 766.163641] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 766.286636] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 766.293347] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 766.301167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 769.036276] Key type ._llcrypt registered [ 769.040241] Key type .llcrypt registered [ 769.213893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 769.237944] alg: No test for adler32 (adler32-zlib) [ 770.267191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.376986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 770.409568] LNet: Added LNI 192.168.201.59@tcp [8/512/0/180] [ 770.414594] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 770.418078] LNet: 52717:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 770.539970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.563729] LNet: 52737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.573131] LNetError: 52737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 770.579321] LNet: Removed LNI 10.1.2.3@tcp [ 770.581513] LNetError: 52737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.597588] LNet: Removed LNI 192.168.201.59@tcp [ 770.610700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.674545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 770.698931] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 770.756786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 770.772505] LNet: 52761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.777248] LNetError: 52761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.795115] LNet: Removed LNI 192.168.201.59@tcp [ 770.816969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.924114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 770.963799] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 770.970716] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 770.976034] LNet: 52769:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 771.134693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.161601] LNet: 52787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.169388] LNetError: 52787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 771.176691] LNet: Removed LNI 10.1.2.3@tcp [ 771.181595] LNetError: 52787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.202590] LNet: Removed LNI 192.168.201.59@tcp [ 771.247909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.359815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 771.391028] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 771.397886] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 771.402189] LNet: 52795:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 771.615604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.631316] LNet: 52821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.636847] LNetError: 52821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 771.645774] LNet: Removed LNI 10.1.2.3@tcp [ 771.650185] LNetError: 52821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.661115] LNet: Removed LNI 192.168.201.59@tcp [ 771.679900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.746717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 771.765619] LNet: Added LNI 192.168.201.59@tcp [8/256/0/182] [ 771.770882] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 771.948717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.972131] LNet: 52849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.978150] LNetError: 52849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 771.983287] LNet: Removed LNI 10.1.2.3@tcp [ 771.988834] LNetError: 52849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.001740] LNet: Removed LNI 192.168.201.59@tcp [ 772.021597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.132808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 772.178054] LNet: Added LNI 192.168.201.59@tcp [8/256/0/182] [ 772.192640] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 772.197158] LNet: 52857:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 772.328722] LNet: Removed LNI 10.1.2.3@tcp [ 772.339142] LNet: Removed LNI 192.168.201.59@tcp [ 773.246944] Key type .llcrypt unregistered [ 773.249247] Key type ._llcrypt unregistered [ 785.530814] Key type ._llcrypt registered [ 785.533196] Key type .llcrypt registered [ 786.918317] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 16:32:20 (1786480340) [ 788.407164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 790.434502] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:32:24 (1786480344) [ 790.982542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 790.998933] alg: No test for adler32 (adler32-zlib) [ 792.023369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.121289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 792.179678] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 793.092369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.59 [ 793.108514] LNet: 54126:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 793.118316] LNet: Removed LNI 192.168.201.59@tcp [ 799.229258] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:32:33 (1786480353) [ 799.502843] LNet: 54696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.508223] LNetError: 54696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.519361] LNet: Removed LNI 192.168.201.59@tcp [ 800.123513] Key type .llcrypt unregistered [ 800.125082] Key type ._llcrypt unregistered [ 807.260413] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 807.511334] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 809.169829] Key type ._llcrypt registered [ 809.172394] Key type .llcrypt registered [ 809.444606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 809.470651] alg: No test for adler32 (adler32-zlib) [ 810.912087] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.388685] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:32:52 (1786480372) [ 818.804641] LNet: 55606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.812815] LNetError: 55606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 818.825006] LNet: Removed LNI 10.1.2.3@tcp [ 819.627154] Key type .llcrypt unregistered [ 819.628875] Key type ._llcrypt unregistered [ 827.498582] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 827.772796] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 827.783756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 827.798766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 830.657968] Key type ._llcrypt registered [ 830.664160] Key type .llcrypt registered [ 830.925397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 830.946469] alg: No test for adler32 (adler32-zlib) [ 832.137558] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 839.950121] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:33:13 (1786480393) [ 841.555290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 843.584431] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:33:17 (1786480397) [ 843.839648] LNet: 56795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.846273] LNetError: 56795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 843.861752] LNet: Removed LNI 10.1.2.3@tcp [ 844.608654] Key type .llcrypt unregistered [ 844.611772] Key type ._llcrypt unregistered [ 852.524553] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 852.796863] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 852.802776] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 852.805262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 855.754910] Key type ._llcrypt registered [ 855.757504] Key type .llcrypt registered [ 855.941287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 855.965979] alg: No test for adler32 (adler32-zlib) [ 856.907701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.959529] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 865.850657] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:33:39 (1786480419) [ 866.400415] LNet: 58030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.408277] LNetError: 58030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 866.432639] LNet: Removed LNI 10.1.2.3@tcp [ 867.075643] Key type .llcrypt unregistered [ 867.078900] Key type ._llcrypt unregistered [ 872.537932] Key type ._llcrypt registered [ 872.541304] Key type .llcrypt registered [ 872.780561] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 872.795118] alg: No test for adler32 (adler32-zlib) [ 873.693887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 873.720850] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 881.017424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 881.724746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 881.751816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 881.790414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 881.822796] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 881.952042] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 882.098364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 882.158714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 882.202627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 882.231332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 882.256505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 882.293962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 882.332332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 882.385130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 882.408420] LNet: 58471:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 882.428533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 882.463800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 882.487215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 882.515819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 882.562699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 882.618515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 882.674153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 883.149792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 883.184044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 883.219801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 883.240659] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 883.446378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 883.470833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 883.500827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 883.520217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 883.540855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 883.575234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 883.613852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 883.647500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 883.667400] LNet: 58580:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 883.684760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 883.735374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 883.799259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 883.841391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 883.886062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 883.922985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 883.976903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 884.450214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 884.489330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 884.559618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 884.577896] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 884.782103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 884.812253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 884.843972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 884.896767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 884.943927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 884.995511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 885.048878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 885.082248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 885.104280] LNet: 58690:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 885.133081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 885.177739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 885.203917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 885.251953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 885.304785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 885.330903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 885.363201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 885.859665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 885.890512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 885.921499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 886.156773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 886.203775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 886.231647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 886.261914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 886.294662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 886.312840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 886.338507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 886.374303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 886.412732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 886.457121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 886.489766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 886.545224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 886.591640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 886.617583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 886.650600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 887.190552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 887.230921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 887.256390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 887.267729] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 887.274272] LNet: Skipped 2 previous similar messages [ 887.524717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 887.573371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 887.630489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 887.678769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 887.760036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 887.804371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 887.846272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 887.890538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 887.907304] LNet: 58908:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 887.912449] LNet: 58908:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 887.925906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 887.958230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 887.982616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 888.012851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 888.037955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 888.071525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 888.118894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 888.741496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 888.788644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 888.830355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 889.087490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 889.111762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 889.140248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 889.166795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 889.196474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 889.231110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 889.258282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 889.284164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 889.307808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 889.346483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 889.382224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 889.419592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 889.454666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 889.484377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 889.529715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 889.897591] LNet: 59063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.901083] LNetError: 59063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.924170] LNet: Removed LNI 192.168.201.59@tcp [ 890.295578] Key type .llcrypt unregistered [ 890.296730] Key type ._llcrypt unregistered [ 907.392054] Key type ._llcrypt registered [ 907.394976] Key type .llcrypt registered [ 908.990354] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:34:22 (1786480462) [ 910.040393] Key type .llcrypt unregistered [ 910.043405] Key type ._llcrypt unregistered [ 917.906620] Key type ._llcrypt registered [ 917.908579] Key type .llcrypt registered [ 918.182416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.216143] alg: No test for adler32 (adler32-zlib) [ 919.156349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 919.190496] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 925.639718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 934.797901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.863092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 934.890366] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 936.462566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 936.520731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 1 [ 936.557183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 936.589234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 1 [ 936.627963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 936.664995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 1 [ 936.698454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 936.727672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 1 [ 936.775427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 936.793352] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 936.796821] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 937.109673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 937.156698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 937.217652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 937.274516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 937.316982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 937.372781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 937.402512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 937.443223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 937.490292] LNet: 60537:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 937.548093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 937.591394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 937.641262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 937.670357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 937.714578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 937.766431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 937.819420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 938.399981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 938.451182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 1 [ 938.488944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 938.528688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 1 [ 938.578622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 938.662197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 1 [ 938.753931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 938.807949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 1 [ 938.866566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 938.888647] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 938.899568] LNet: Skipped 2 previous similar messages [ 939.168086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 939.207604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 939.250900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 939.290962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 939.324685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 939.354201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 939.386391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 939.410896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 939.424749] LNet: 60659:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 939.445737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 939.476393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 939.515822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 939.584531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 939.639982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 939.679423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 939.717704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 940.386091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 940.434932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 1 [ 940.488844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 940.538849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 1 [ 940.574209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 940.622539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 1 [ 940.678428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 940.724729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 1 [ 940.774972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 940.804852] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 940.810849] LNet: Skipped 2 previous similar messages [ 941.131834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 941.172376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 941.208271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 941.231137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 941.283080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 941.332274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 941.391523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 941.440410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 941.457865] LNet: 60781:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 941.484447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 941.524313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 941.584455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 941.639830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 941.698764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 941.753384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 941.809424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 942.327824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 942.376906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 1 [ 942.415881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 942.481079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 1 [ 942.525355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 942.575500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 1 [ 942.627389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 942.700793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 1 [ 942.738276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 943.016572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 943.055755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 943.087549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 943.126900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 943.166606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 943.208033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 943.255883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 943.300981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 943.340597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 943.383648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 943.427388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 943.469922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 943.539690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 943.571669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 943.634229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 944.292857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 944.328512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 1 [ 944.364945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 944.407098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 1 [ 944.457765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 944.489694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 1 [ 944.530145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 944.565973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 1 [ 944.604241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 944.621923] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 944.625877] LNet: Skipped 5 previous similar messages [ 944.861644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 944.885740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 944.911491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 944.934283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 944.958908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 944.984754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 945.007255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 945.039242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 945.047537] LNet: 61023:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 945.055129] LNet: 61023:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 945.067300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 945.087170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 945.110774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 945.134202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 945.161614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 945.186258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 945.207614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 945.618205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 945.648184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_error -r 1 [ 945.677200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 945.716085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_error -r 1 [ 945.757682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 945.793171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_error -r 1 [ 945.821526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 945.856980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_error -r 1 [ 945.891568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 946.139086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 946.173874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 946.203476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 946.235607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 946.263980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 946.301708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 946.354337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 946.385933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 946.418701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 946.448625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 946.481712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 946.513582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 946.539646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 946.585634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 946.623088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 951.555231] LNet: 61225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.560380] LNetError: 61225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.608824] LNet: Removed LNI 192.168.201.59@tcp [ 952.613178] LNetError: 61225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.633773] LNet: Removed LNI 192.168.201.59@tcp1 [ 952.947816] Key type .llcrypt unregistered [ 952.950638] Key type ._llcrypt unregistered [ 966.759540] Key type ._llcrypt registered [ 966.760970] Key type .llcrypt registered [ 967.865301] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:35:21 (1786480521) [ 968.807366] Key type .llcrypt unregistered [ 968.813165] Key type ._llcrypt unregistered [ 975.303936] Key type ._llcrypt registered [ 975.310893] Key type .llcrypt registered [ 975.522703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.545756] alg: No test for adler32 (adler32-zlib) [ 976.507892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 976.534647] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 983.919168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 984.740502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 984.794383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 984.843686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 984.869309] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 985.160701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 985.191945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 985.224608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 985.256709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 985.293155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 985.325871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 985.365546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 985.418388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 985.448499] LNet: 62476:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 985.470653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 985.508460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 985.555642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 985.587680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 985.632488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 985.667176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 985.695965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 986.398124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 986.436127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 986.482230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 986.517386] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 986.768403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 986.802945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 986.858183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 986.887853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 986.929436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 986.954508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 986.981387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 987.001424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 987.022787] LNet: 62585:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 987.038939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 987.085130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 987.114208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 987.139197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 987.164926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 987.222155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 987.282845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 987.910530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 987.943535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 987.973459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 987.995221] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 988.233878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 988.257443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 988.287603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 988.317538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 988.346207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 988.372227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 988.402589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 988.439820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 988.464482] LNet: 62695:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 988.481172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 988.504070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 988.535268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 988.558510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 988.595934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.633154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 988.662793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 988.926419] LNet: 62740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.932766] LNetError: 62740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.991470] LNet: Removed LNI 192.168.201.59@tcp [ 990.453512] Key type .llcrypt unregistered [ 990.460894] Key type ._llcrypt unregistered [ 1008.054144] Key type ._llcrypt registered [ 1008.057978] Key type .llcrypt registered [ 1009.277876] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:36:03 (1786480563) [ 1010.130556] Key type .llcrypt unregistered [ 1010.132914] Key type ._llcrypt unregistered [ 1016.291430] Key type ._llcrypt registered [ 1016.294683] Key type .llcrypt registered [ 1016.494462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1016.518571] alg: No test for adler32 (adler32-zlib) [ 1017.450566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1017.481432] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1024.285247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1032.010714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1032.052954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1032.069946] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1033.329703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1033.370035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 1 [ 1033.406300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1033.455584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 1 [ 1033.488316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1033.540586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 1 [ 1033.568815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1033.598418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 1 [ 1033.635743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1033.649483] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1033.654489] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 1033.832769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.857793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1033.884362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.910837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.940946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.969528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1033.998645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.025490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.040438] LNet: 64211:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.053901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.078234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.103931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.134388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.171173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.195330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.231185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1034.263958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.867936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1034.912252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_error -r 1 [ 1034.954621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1035.008889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_error -r 1 [ 1035.059643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1035.112602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_error -r 1 [ 1035.171491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1035.225541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_error -r 1 [ 1035.274420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1035.292622] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1035.295646] LNet: Skipped 2 previous similar messages [ 1035.539582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.586860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1035.631186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.671143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.743525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.800502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.849761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.876366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.888909] LNet: 64335:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1035.903606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.931908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.968968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.996524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1036.025974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.069159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.124532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1036.155295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.709671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1036.770372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_timeout -r 1 [ 1036.807271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1036.894511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_timeout -r 1 [ 1036.942986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1036.986355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_timeout -r 1 [ 1037.031884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1037.072858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_timeout -r 1 [ 1037.150849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1037.193256] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1037.492496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1037.527891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1037.565748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1037.595349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1037.626422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1037.659733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1037.685112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1037.722696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1037.746650] LNet: 64458:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1037.775281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1037.805653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1037.834696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1037.870507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1037.900333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1037.931373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1037.961854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1038.007685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1042.795842] LNet: 64540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.810694] LNetError: 64540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.822266] LNet: Removed LNI 192.168.201.59@tcp [ 1042.838336] LNet: Removed LNI 192.168.201.59@tcp1 [ 1043.214931] Key type .llcrypt unregistered [ 1043.216592] Key type ._llcrypt unregistered [ 1058.136223] Key type ._llcrypt registered [ 1058.137618] Key type .llcrypt registered [ 1059.236431] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:36:53 (1786480613) [ 1060.346678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1061.990554] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:36:56 (1786480616) [ 1062.877724] Key type .llcrypt unregistered [ 1062.889744] Key type ._llcrypt unregistered [ 1067.834495] Key type ._llcrypt registered [ 1067.837589] Key type .llcrypt registered [ 1068.057398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1068.090121] alg: No test for adler32 (adler32-zlib) [ 1069.003344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1069.042942] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1075.470404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1076.058917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1076.079160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1076.114813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1076.134103] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1076.255819] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1076.335536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.368788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1076.392727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.419668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.447206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.468270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.490760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.512174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.527960] LNet: 66087:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1076.545968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1076.585674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1076.618481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1076.649489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1076.686392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.715914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.089655] LNet: 66130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.094475] LNetError: 66130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.116362] LNet: Removed LNI 192.168.201.59@tcp [ 1077.467368] Key type .llcrypt unregistered [ 1077.468933] Key type ._llcrypt unregistered [ 1093.750387] Key type ._llcrypt registered [ 1093.755916] Key type .llcrypt registered [ 1094.013305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1094.050459] alg: No test for adler32 (adler32-zlib) [ 1094.950246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1094.976672] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1102.095799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1111.422630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.470992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1111.520534] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1112.814365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1112.850326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e network_timeout -r 1 [ 1112.889177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1112.923611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e network_timeout -r 1 [ 1112.965918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1112.999938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e network_timeout -r 1 [ 1113.044828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1113.090207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e network_timeout -r 1 [ 1113.128395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1113.148325] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1113.373791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.424950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1113.442849] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1113.451039] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 1113.470440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.526645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.558405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.587280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.622597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.652651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.664325] LNet: 67013:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.684157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.721575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.766648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.802751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.833451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.860563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.269379] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1119.625995] LNet: 67090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.632624] LNetError: 67090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.642251] LNet: Removed LNI 192.168.201.59@tcp [ 1119.660270] LNet: Removed LNI 192.168.201.59@tcp1 [ 1120.063591] Key type .llcrypt unregistered [ 1120.069417] Key type ._llcrypt unregistered [ 1137.615371] Key type ._llcrypt registered [ 1137.617928] Key type .llcrypt registered [ 1138.962561] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:38:12 (1786480692) [ 1139.377503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1139.393825] alg: No test for adler32 (adler32-zlib) [ 1140.357487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1140.444866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1140.477564] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1141.314656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1141.338665] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1141.984983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1142.063791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1142.126343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1142.162760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1142.223613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1142.264601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1142.306835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1142.324646] LNetError: 68190:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1142.495794] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1142.495867] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1144.543698] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1144.824794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1148.639374] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1149.489570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1149.515352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1149.593143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1149.615105] LNet: 68297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.621297] LNetError: 68297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.628792] LNet: Removed LNI 192.168.201.59@tcp [ 1149.639294] LNet: Removed LNI 192.168.201.59@tcp1 [ 1149.673381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1149.755991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1149.812081] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1150.438685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1150.459370] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1151.249352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1151.297689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.362982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1151.415563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1151.474395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1151.525699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1151.576184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1151.610779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1151.640574] LNetError: 68569:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1151.780160] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1154.143297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1157.920184] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1157.928495] LNet: Skipped 1 previous similar message [ 1162.165915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1162.208851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1162.250749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.649367] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:38:42 (1786480722) [ 1168.885442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1168.897744] LNet: 69277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.900680] LNetError: 69277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.903989] LNetError: 69277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1168.906851] LNet: Removed LNI 192.168.201.59@tcp [ 1168.913861] LNet: Removed LNI 192.168.201.59@tcp1 [ 1168.934078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1169.028690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1169.057108] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1170.101375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1170.125545] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1171.121937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1171.193252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1171.222326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1171.252434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1171.280549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1171.308122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1171.331605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 0 [ 1171.442817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1172.063568] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.59@tcp [ 1172.066667] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1172.081659] LNet: Skipped 1 previous similar message [ 1186.901294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1187.081867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1187.116634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.160432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 500 [ 1187.234693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1193.850572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1193.898102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1193.920516] LNet: 69769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.929787] LNetError: 69769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1193.940903] LNetError: 69769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1193.944039] LNet: Removed LNI 192.168.201.59@tcp [ 1193.955506] LNet: Removed LNI 192.168.201.59@tcp1 [ 1193.978263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1194.045327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1194.076472] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1194.656554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1194.692897] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1195.282392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1195.330825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.396939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1195.422441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1195.473516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1195.533727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1195.573729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1195.609194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 0 [ 1196.064136] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1196.070722] LNet: Skipped 3 previous similar messages [ 1196.920514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1207.107822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1207.162982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1207.222811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.908416] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:39:26 (1786480766) [ 1213.372428] LNet: 70815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.404798] LNetError: 70815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1213.414487] LNetError: 70815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1213.422634] LNet: Removed LNI 192.168.201.59@tcp [ 1213.432861] LNet: Removed LNI 192.168.201.59@tcp1 [ 1214.075835] Key type .llcrypt unregistered [ 1214.077651] Key type ._llcrypt unregistered [ 1219.130443] Key type ._llcrypt registered [ 1219.132016] Key type .llcrypt registered [ 1219.330777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1219.346859] alg: No test for adler32 (adler32-zlib) [ 1220.245732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1220.276356] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1228.145159] Lustre: DEBUG MARKER: Initial discovery [ 1228.202707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1230.114586] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1230.157275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1230.207526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1230.254323] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1230.296917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1230.324625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.159@tcp -r 1 -m GET -l 3 [ 1230.357324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.159@tcp -r 1 -m GET -e local_error [ 1230.396092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.159@tcp -r 1 -m PUT -l 6 [ 1230.562043] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1231.623226] Lustre: DEBUG MARKER: Force oleg159-client.virtnet to discover 192.168.201.159@tcp (in background) [ 1231.691058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1232.777785] Lustre: DEBUG MARKER: Force oleg159-server to discover 192.168.201.59@tcp [ 1234.677632] Lustre: DEBUG MARKER: Wait for 71572 [ 1234.789161] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1235.870693] Lustre: DEBUG MARKER: Finished wait on 71572 [ 1236.013243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1236.038908] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1236.055965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.279617] LNet: 72035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.285737] LNetError: 72035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.302943] LNet: Removed LNI 192.168.201.59@tcp [ 1236.608220] Key type .llcrypt unregistered [ 1236.609699] Key type ._llcrypt unregistered [ 1250.572300] Key type ._llcrypt registered [ 1250.573878] Key type .llcrypt registered [ 1252.004825] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:40:05 (1786480805) [ 1252.427735] Key type .llcrypt unregistered [ 1252.429953] Key type ._llcrypt unregistered [ 1257.417518] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1257.578716] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1257.583308] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1257.586626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1260.038753] Key type ._llcrypt registered [ 1260.043435] Key type .llcrypt registered [ 1260.241785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1260.264185] alg: No test for adler32 (adler32-zlib) [ 1261.236254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1261.282925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1261.305750] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1262.119510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1262.136550] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1270.268650] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:40:24 (1786480824) [ 1270.599990] LNet: 74117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1270.608854] LNetError: 74117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1270.615804] LNet: Removed LNI 10.1.2.3@tcp [ 1270.632205] LNet: Removed LNI 192.168.201.59@tcp [ 1271.292845] Key type .llcrypt unregistered [ 1271.296421] Key type ._llcrypt unregistered [ 1277.381274] Key type ._llcrypt registered [ 1277.382944] Key type .llcrypt registered [ 1277.603598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1277.623642] alg: No test for adler32 (adler32-zlib) [ 1278.592817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.641395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1278.668691] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1279.241695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1279.256417] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1280.011271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.59@tcp [ 1281.058898] LNetError: Unexpected error -22 connecting to 192.168.201.59@tcp at host 192.168.201.59:988 [ 1285.301151] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:40:39 (1786480839) [ 1285.445202] LNet: 75249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.453943] LNetError: 75249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1285.458955] LNet: Removed LNI 10.1.2.3@tcp [ 1285.478106] LNet: Removed LNI 192.168.201.59@tcp [ 1285.979559] Key type .llcrypt unregistered [ 1285.982465] Key type ._llcrypt unregistered [ 1291.598977] Key type ._llcrypt registered [ 1291.601533] Key type .llcrypt registered [ 1291.769925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1291.778126] alg: No test for adler32 (adler32-zlib) [ 1292.726560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.794176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1292.825831] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1293.492869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1293.515261] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1294.251736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.59@tcp1 --nid 192.168.201.59@tcp2 [ 1297.602890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp1 192.168.201.59@tcp2 [ 1297.619848] LNetError: 76091:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.59@tcp1 to 192.168.201.59@tcp2 - Route table may be misconfigured [ 1297.657964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp2 192.168.201.59@tcp1 [ 1302.828733] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:40:56 (1786480856) [ 1303.067232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1303.086228] LNet: 76664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.090218] LNetError: 76664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.093993] LNet: Removed LNI 192.168.201.59@tcp1 [ 1304.170623] LNet: Removed LNI 192.168.201.59@tcp2 [ 1304.191656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.233662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1304.252385] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1304.897974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1304.914295] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1305.666222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1305.703656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.59@tcp [ 1305.763091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp1 192.168.201.59@tcp1 [ 1305.803771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.59@tcp -d 192.168.201.59@tcp -e network_timeout [ 1305.836601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -e network_timeout [ 1305.888697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -e network_timeout [ 1305.951793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -e network_timeout [ 1305.981835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1305.999371] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1306.024993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1306.118339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1306.192188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1306.277282] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1311.697224] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:41:05 (1786480865) [ 1311.795033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1311.820641] LNet: 77519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.826734] LNetError: 77519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.834772] LNetError: 77519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1311.844936] LNet: Removed LNI 192.168.201.59@tcp [ 1312.874526] LNet: Removed LNI 192.168.201.59@tcp1 [ 1312.930883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1313.009413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1313.282151] LNet: 77566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.614321] Key type .llcrypt unregistered [ 1313.622636] Key type ._llcrypt unregistered [ 1323.301418] Key type ._llcrypt registered [ 1323.303168] Key type .llcrypt registered [ 1324.336161] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:41:18 (1786480878) [ 1324.674992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1324.696839] alg: No test for adler32 (adler32-zlib) [ 1325.577799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1326.643181] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1328.090429] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:41:22 (1786480882) [ 1328.227822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1328.258808] LNet: 78663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.285608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.359841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1328.390878] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1328.991846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1329.011350] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1329.645211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1329.675195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp1 [ 1329.705735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp1 [ 1334.721572] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:41:28 (1786480888) [ 1336.385726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg159-server [ 1337.762138] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:41:31 (1786480891) [ 1339.362344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg159-server [ 1340.835596] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:41:35 (1786480895) [ 1342.376883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg159-server [ 1343.797612] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:41:38 (1786480898) [ 1345.397830] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg159-server [ 1346.743464] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:41:40 (1786480900) [ 1348.121656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg159-server [ 1349.429763] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:41:43 (1786480903) [ 1350.919666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg159-server [ 1352.314349] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:41:46 (1786480906) [ 1353.756721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg159-server [ 1354.963672] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:41:49 (1786480909) [ 1356.398471] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg159-server [ 1357.792676] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:41:52 (1786480912) [ 1359.418250] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg159-server [ 1361.003250] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:41:55 (1786480915) [ 1361.236745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1361.257943] LNet: 82540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.261923] LNetError: 82540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.271170] LNet: Removed LNI 192.168.201.59@tcp [ 1362.275519] LNetError: 82540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.304815] LNet: Removed LNI 192.168.201.59@tcp1 [ 1362.346904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.427103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.468903] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1363.053764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1363.123711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1363.251901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1363.270803] LNet: 82687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1363.279848] LNetError: 82687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1364.328906] LNet: Removed LNI 192.168.201.59@tcp [ 1364.352294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.409636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1364.435691] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1365.005905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1365.197176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1366.247340] LNet: Removed LNI 192.168.201.59@tcp [ 1366.298783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.388749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1366.431056] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1367.038683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1367.090239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1367.244257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1367.259951] LNet: 82980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.263820] LNet: 82980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1367.268450] LNetError: 82980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.274532] LNetError: 82980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1367.286176] LNet: Removed LNI 192.168.201.59@tcp [ 1367.302425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1367.351340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1367.381703] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1367.928701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1368.064548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1369.136839] LNet: Removed LNI 192.168.201.59@tcp [ 1369.163542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.214842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1369.239799] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1369.716638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1369.767861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1369.916458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.998341] LNet: Removed LNI 192.168.201.59@tcp [ 1371.046285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.107588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1371.140039] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1371.871605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1372.049871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1372.059796] LNet: 83418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1372.062892] LNet: 83418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1372.067182] LNetError: 83418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.069469] LNetError: 83418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1372.098123] LNet: Removed LNI 192.168.201.59@tcp [ 1372.127733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.165856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1372.184818] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1372.918798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1372.975478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1373.129475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1374.187752] LNet: Removed LNI 192.168.201.59@tcp [ 1374.222292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.269660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1374.282453] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1374.871805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1375.010558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1376.043651] LNet: Removed LNI 192.168.201.59@tcp [ 1376.070727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.167927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1376.193442] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1376.704897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1376.750271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1376.976940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1378.084791] LNet: Removed LNI 192.168.201.59@tcp [ 1378.101713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.189536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1378.223582] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1378.875550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1379.105882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1380.134569] LNet: Removed LNI 192.168.201.59@tcp [ 1380.177657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.230975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1380.254826] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1380.900360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1380.959282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1381.206580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.245975] LNet: 84150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1381.251424] LNet: 84150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1381.256963] LNetError: 84150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1381.264993] LNetError: 84150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1381.278611] LNet: Removed LNI 192.168.201.59@tcp [ 1381.293846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.336791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1381.364082] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1381.939216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1383.265157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1384.361927] LNet: Removed LNI 192.168.201.59@tcp [ 1384.394182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1384.486855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1384.509994] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1385.106893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1385.166608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1385.202277] LNet: Removed LNI 192.168.201.59@tcp [ 1385.227949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1385.287945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1385.309312] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1386.064346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1386.145868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1386.169524] LNet: Removed LNI 192.168.201.59@tcp [ 1386.183237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.235653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1386.315525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1386.348085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.404550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1386.475935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1386.518303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.559461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1386.629389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1386.665979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.711175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1386.742733] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1387.432450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.457823] LNet: Removed LNI 192.168.201.59@tcp [ 1387.474615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.592347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1393.247442] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:42:27 (1786480947) [ 1393.457194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1393.483860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.521190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1393.543366] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1394.525583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1394.588432] LNet: Removed LNI 192.168.201.59@tcp231 [ 1394.608441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.706420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1394.757233] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1394.763513] LNet: 85472:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1395.186216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1395.222718] LNet: Removed LNI 192.168.201.59@tcp231 [ 1395.247424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1395.280113] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1396.314438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1396.335928] LNet: Removed LNI 192.168.201.59@tcp231 [ 1396.349914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.402453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1396.427143] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/0] [ 1396.432293] LNet: 85624:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1396.811387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1396.835779] LNet: Removed LNI 192.168.201.59@tcp231 [ 1396.848142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1396.875373] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/0] [ 1397.861340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1397.884400] LNet: 85767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.887426] LNet: 85767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1397.903981] LNetError: 85767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.911956] LNetError: 85767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1397.934955] LNet: Removed LNI 192.168.201.59@tcp231 [ 1397.973288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1398.072249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1398.117477] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/60] [ 1398.127860] LNet: 85775:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1398.561965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1398.604916] LNet: Removed LNI 192.168.201.59@tcp231 [ 1398.627354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1398.673268] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/60] [ 1399.747168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.768925] LNet: Removed LNI 192.168.201.59@tcp231 [ 1399.786271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.834394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1399.851037] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1400.405922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1400.442158] LNet: Removed LNI 192.168.201.59@tcp231 [ 1400.462440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1400.497601] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1407.347748] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:42:41 (1786480961) [ 1407.573720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1407.601764] LNet: Removed LNI 192.168.201.59@tcp231 [ 1407.622714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.680670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1407.712401] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1408.210358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1408.244600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1408.267605] LNet: Removed LNI 192.168.201.59@tcp232 [ 1408.281463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1408.319275] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1408.936528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1408.963740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1408.990301] LNet: Removed LNI 192.168.201.59@tcp232 [ 1409.001724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1409.035049] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1409.771849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1409.799248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1409.821855] LNet: Removed LNI 192.168.201.59@tcp232 [ 1409.839197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1409.873952] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1410.755228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1410.805631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1410.835205] LNet: Removed LNI 192.168.201.59@tcp232 [ 1410.865226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1417.083584] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:42:51 (1786480971) [ 1418.287312] Key type .llcrypt unregistered [ 1418.292421] Key type ._llcrypt unregistered [ 1424.512546] Key type ._llcrypt registered [ 1424.514507] Key type .llcrypt registered [ 1424.740380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1424.748124] alg: No test for adler32 (adler32-zlib) [ 1425.647580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1425.667309] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1433.392283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1441.417827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.474079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1441.510098] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1442.815314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 2 [ 1442.846217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 2 [ 1442.879697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 2 [ 1442.915684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 2 [ 1442.953900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_interrupt -r 2 [ 1442.999958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 2 [ 1443.039147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_interrupt -r 2 [ 1443.077117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 2 [ 1443.147537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1443.168502] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1443.173390] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1443.367112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1443.397690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1443.417092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1443.441253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1443.465750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1443.490630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1443.520192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1443.546447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1443.556569] LNet: 88431:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1443.572203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1443.611184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1443.639817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1443.673615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1443.715393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1443.745476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1444.221716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 2 [ 1444.248906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 2 [ 1444.279456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 2 [ 1444.330868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 2 [ 1444.373609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_dropped -r 2 [ 1444.409508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 2 [ 1444.457413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_dropped -r 2 [ 1444.503311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 2 [ 1444.532134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1444.550384] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1444.764662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1444.789624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1444.815365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1444.844254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1444.866665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1444.881966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1444.896345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1444.910330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1444.920567] LNet: 88551:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1444.935253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1444.970428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1445.006876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1445.036553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1445.064573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1445.116580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1445.604364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 2 [ 1445.631238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 2 [ 1445.667579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 2 [ 1445.696345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 2 [ 1445.728766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_aborted -r 2 [ 1445.752034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 2 [ 1445.786263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_aborted -r 2 [ 1445.820969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 2 [ 1445.851126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1445.872725] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1446.104479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1446.130000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1446.158972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1446.188407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1446.223537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1446.260564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1446.290584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1446.331422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1446.350629] LNet: 88671:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1446.377498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1446.449401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1446.488885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1446.518681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1446.555034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1446.590572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1447.085382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 2 [ 1447.124622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 2 [ 1447.170551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 2 [ 1447.212650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 2 [ 1447.259601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_no_route -r 2 [ 1447.302651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 2 [ 1447.336692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_no_route -r 2 [ 1447.394149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 2 [ 1447.441788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1447.747182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1447.779818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1447.828230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1447.872228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1447.900606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1447.954883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1448.000017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1448.054959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1448.115374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1448.159598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1448.215653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1448.272138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1448.332898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1448.374982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1449.054299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 2 [ 1449.097691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 2 [ 1449.143494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 2 [ 1449.180506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 2 [ 1449.225626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_timeout -r 2 [ 1449.322855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 2 [ 1449.397995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_timeout -r 2 [ 1449.445980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 2 [ 1449.494180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1449.513172] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1449.517155] LNet: Skipped 1 previous similar message [ 1449.820356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1449.858060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1449.878475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1449.905423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1449.940418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1449.978232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1450.017400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1450.068829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1450.113165] LNet: 88912:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1450.121403] LNet: 88912:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1450.169718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1450.211414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1450.250795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1450.286931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1450.325083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1450.369598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1450.839472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 2 [ 1450.895561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 2 [ 1450.946806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 2 [ 1451.026542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 2 [ 1451.119348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_dropped -r 2 [ 1451.168906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 2 [ 1451.214984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_dropped -r 2 [ 1451.271321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 2 [ 1451.338842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1451.372054] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 1451.722162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1451.762285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1451.790801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1451.829123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1451.857942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1451.888908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1451.921919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1451.965735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1452.048957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1452.082451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1452.122085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1452.157803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1452.212378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1452.241921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1457.218199] LNet: 89110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.224370] LNetError: 89110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1457.229265] LNet: Removed LNI 192.168.201.59@tcp [ 1457.237619] LNet: Removed LNI 192.168.201.59@tcp1 [ 1457.540695] Key type .llcrypt unregistered [ 1457.542136] Key type ._llcrypt unregistered [ 1471.398686] Key type ._llcrypt registered [ 1471.399954] Key type .llcrypt registered [ 1472.510358] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 16:43:46 (1786481026) [ 1472.955537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1472.972665] alg: No test for adler32 (adler32-zlib) [ 1473.913295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.973796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1473.999348] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1474.894515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1474.929893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1474.950597] LNetError: 90089:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1474.976412] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1475.085986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.59@tcp [ 1475.191707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1481.141843] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 16:43:55 (1786481035) [ 1481.221641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.242787] LNet: 90673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.256895] LNetError: 90673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.275742] LNet: Removed LNI 192.168.201.59@tcp [ 1481.306148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.402893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.443490] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1482.004325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1482.015414] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1482.725684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.59@tcp [ 1482.737711] LNetError: 90945:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1483.426985] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1483.911341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.59@tcp [ 1491.187716] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 16:44:05 (1786481045) [ 1493.004660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg159-server [ 1494.350368] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 16:44:08 (1786481048) [ 1495.882989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg159-server [ 1497.553602] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 16:44:11 (1786481051) [ 1499.400783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg159-server [ 1501.212775] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:44:15 (1786481055) [ 1501.487759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.510623] LNet: 92576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.519474] LNetError: 92576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.527403] LNet: Removed LNI 192.168.201.59@tcp [ 1501.540430] LNet: Removed LNI 192.168.201.59@tcp1 [ 1501.564384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.612982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.628044] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1502.114970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1502.174451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1502.223355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1502.285583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1502.352759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1502.397698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1508.167506] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:44:22 (1786481062) [ 1508.228467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.250099] LNet: 93294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.257974] LNetError: 93294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.267279] LNetError: 93294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1508.282885] LNet: Removed LNI 192.168.201.59@tcp [ 1508.300417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.355824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.376977] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1515.578431] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:44:29 (1786481069) [ 1515.707158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1515.727746] LNet: 93999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.733820] LNetError: 93999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.765038] LNet: Removed LNI 192.168.201.59@tcp [ 1515.791217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.851750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1515.888374] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1522.069199] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:44:36 (1786481076) [ 1523.359544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1525.038275] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:44:39 (1786481079) [ 1525.666626] LNet: 95039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.672717] LNetError: 95039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.689230] LNet: Removed LNI 192.168.201.59@tcp [ 1526.145119] Key type .llcrypt unregistered [ 1526.146568] Key type ._llcrypt unregistered [ 1531.219655] Key type ._llcrypt registered [ 1531.228299] Key type .llcrypt registered [ 1531.435508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1531.453561] alg: No test for adler32 (adler32-zlib) [ 1532.400275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1532.425366] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1538.932476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1543.520960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.159@tcp [ 1543.827680] LNet: 95464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.842127] LNetError: 95464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1543.858811] LNet: Removed LNI 192.168.201.59@tcp [ 1544.151713] Key type .llcrypt unregistered [ 1544.154740] Key type ._llcrypt unregistered [ 1553.407257] Key type ._llcrypt registered [ 1553.410491] Key type .llcrypt registered [ 1554.611390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1556.041623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1557.442855] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:45:11 (1786481111) [ 1558.502702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1560.029439] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:45:14 (1786481114) [ 1561.619617] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg159-server [ 1563.205457] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:45:17 (1786481117) [ 1564.424930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg159-server [ 1565.965723] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:45:19 (1786481119) [ 1567.238910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1567.263762] alg: No test for adler32 (adler32-zlib) [ 1568.234973] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 1568.398839] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1570.047826] Key type lgssc registered [ 1570.855938] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1585.695193] Key type lgssc unregistered [ 1585.955585] LNet: 98641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.963717] LNetError: 98641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.998238] LNet: Removed LNI 192.168.201.59@tcp [ 1586.469878] Key type .llcrypt unregistered [ 1586.472095] Key type ._llcrypt unregistered [ 1597.771767] Key type ._llcrypt registered [ 1597.773558] Key type .llcrypt registered [ 1598.794277] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:45:52 (1786481152) [ 1599.164136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1599.179101] alg: No test for adler32 (adler32-zlib) [ 1600.045064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.086655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1600.117185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1600.146871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1600.179993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1604.318939] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:45:58 (1786481158) [ 1604.412521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1604.438219] LNet: 100056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.463169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.501942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1604.521249] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1605.065613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.089322] LNet: 100200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.092778] LNetError: 100200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.108718] LNet: Removed LNI 192.168.201.59@tcp [ 1605.123774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.151456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1605.163884] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1605.422992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.448877] LNet: Removed LNI 192.168.201.59@tcp [ 1605.463982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.495895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1605.525248] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1609.833546] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:46:04 (1786481164) [ 1610.071809] LNet: 100818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.075631] LNet: 100818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1610.080531] LNetError: 100818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.083812] LNetError: 100818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1610.095402] LNet: Removed LNI 192.168.201.59@tcp [ 1610.613817] Key type .llcrypt unregistered [ 1610.617302] Key type ._llcrypt unregistered [ 1616.135874] Key type ._llcrypt registered [ 1616.137197] Key type .llcrypt registered [ 1616.289729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1616.299401] alg: No test for adler32 (adler32-zlib) [ 1617.191865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.273266] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1617.276687] LNet: *** cfs_fail_loc=e003, val=100*** [ 1617.777568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1617.790724] LNet: *** cfs_fail_loc=e003, val=100*** [ 1617.792877] LNet: Skipped 2 previous similar messages [ 1617.799417] Ping reply received with mapping: [ 1617.804247] found metadata reply NI[192.168.201.59@tcp] [ 1617.806891] 3b c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1617.809830] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1617.813540] 5a 5a 5a 5a 5a 5a 5a 5a [ 1617.933821] LNet: 101220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.939185] LNetError: 101220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.953200] LNet: Removed LNI 192.168.201.59@tcp [ 1618.450309] Key type .llcrypt unregistered [ 1618.452527] Key type ._llcrypt unregistered [ 1626.854611] Key type ._llcrypt registered [ 1626.857342] Key type .llcrypt registered [ 1627.929928] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:46:22 (1786481182) [ 1628.484391] Key type .llcrypt unregistered [ 1628.486312] Key type ._llcrypt unregistered [ 1634.407720] Key type ._llcrypt registered [ 1634.409492] Key type .llcrypt registered [ 1634.559654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1634.571372] alg: No test for adler32 (adler32-zlib) [ 1635.592648] LNetError: 102277:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1635.887628] Key type .llcrypt unregistered [ 1635.889804] Key type ._llcrypt unregistered [ 1639.904488] Key type ._llcrypt registered [ 1639.905610] Key type .llcrypt registered [ 1640.680549] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:46:35 (1786481195) [ 1641.071870] Key type .llcrypt unregistered [ 1641.073538] Key type ._llcrypt unregistered [ 1648.606230] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1648.622533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1652.049613] Key type ._llcrypt registered [ 1652.056521] Key type .llcrypt registered [ 1652.498851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1652.552092] alg: No test for adler32 (adler32-zlib) [ 1653.573829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.650510] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1665.834878] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:46:59 (1786481219) [ 1666.107291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.166491] LNet: 104066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.174590] LNetError: 104066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1666.197800] LNet: Removed LNI 10.1.2.3@tcp [ 1666.248672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.367062] Key type .llcrypt unregistered [ 1667.370610] Key type ._llcrypt unregistered [ 1677.254148] Key type ._llcrypt registered [ 1677.263866] Key type .llcrypt registered [ 1677.569439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1677.634445] alg: No test for adler32 (adler32-zlib) [ 1678.598035] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1678.723869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1678.978361] LNet: 104361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.997412] LNetError: 104361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.039393] LNet: Removed LNI 192.168.201.59@tcp [ 1681.114793] Key type .llcrypt unregistered [ 1681.118407] Key type ._llcrypt unregistered [ 1697.377177] Key type ._llcrypt registered [ 1697.378469] Key type .llcrypt registered [ 1698.759837] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:47:32 (1786481252) [ 1699.728635] Key type .llcrypt unregistered [ 1699.730329] Key type ._llcrypt unregistered [ 1707.645786] Key type ._llcrypt registered [ 1707.648451] Key type .llcrypt registered [ 1707.833397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1707.864926] alg: No test for adler32 (adler32-zlib) [ 1715.088456] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:47:48 (1786481268) [ 1715.169656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.208926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.290256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.331723] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1715.899738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.923670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1715.945516] LNet: Removed LNI 192.168.201.59@tcp [ 1716.183902] LNet: 106120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.512794] Key type .llcrypt unregistered [ 1716.515493] Key type ._llcrypt unregistered [ 1726.528951] Key type ._llcrypt registered [ 1726.531259] Key type .llcrypt registered [ 1727.698831] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:48:01 (1786481281) [ 1728.115688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1728.129625] alg: No test for adler32 (adler32-zlib) [ 1729.066690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.109863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.128668] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1729.685569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1729.714938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.59@tcp [ 1729.730297] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1729.733625] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1729.736323] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1729, 0, 0 [ 1729.740483] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1729.743342] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1729.746172] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1729.749490] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1729, 0, 0 [ 1729.754554] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1729.757706] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1729.768416] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1729.772493] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1729, 0, 1779 [ 1729.776529] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1729.779340] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1729.782946] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1729.791050] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1729, 0, 1779 [ 1729.796169] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1736.044642] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:48:10 (1786481290) [ 1736.719486] LNet: 107659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.726141] LNetError: 107659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1736.737450] LNet: Removed LNI 192.168.201.59@tcp [ 1737.441392] Key type .llcrypt unregistered [ 1737.443862] Key type ._llcrypt unregistered [ 1743.642575] Key type ._llcrypt registered [ 1743.643979] Key type .llcrypt registered [ 1743.867447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1743.881682] alg: No test for adler32 (adler32-zlib) [ 1744.809494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1744.833593] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1751.819252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1760.621638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.651455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1760.683156] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1761.599110] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1761.770040] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1761.781996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1761.795144] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1764.194834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1764.231740] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1774.133845] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:48:48 (1786481328) [ 1774.314754] LNet: 109171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.323853] LNetError: 109171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.331476] LNet: Removed LNI 192.168.201.59@tcp [ 1775.391769] LNet: Removed LNI 192.168.201.59@tcp1 [ 1775.406380] LNetError: 109171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1775.426310] LNetError: 109171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1775.455347] LNet: Removed LNI 10.1.2.3@tcp99 [ 1776.242561] Key type .llcrypt unregistered [ 1776.245291] Key type ._llcrypt unregistered [ 1782.978975] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1782.987834] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1785.623386] Key type ._llcrypt registered [ 1785.625231] Key type .llcrypt registered [ 1785.782675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1785.800208] alg: No test for adler32 (adler32-zlib) [ 1786.726151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.778695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.813723] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1787.476359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1787.495600] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1789.399914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1789.450528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1789.514561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1789.545118] LNetError: 109920:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1789.592617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1789.657231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1790.106300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1790.190929] LNet: 109940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.208524] LNetError: 109940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1790.222373] LNet: Removed LNI 10.1.2.3@tcp [ 1791.270527] LNet: Removed LNI 192.168.201.59@tcp [ 1792.213695] Key type .llcrypt unregistered [ 1792.217283] Key type ._llcrypt unregistered [ 1804.252785] Key type ._llcrypt registered [ 1804.257616] Key type .llcrypt registered [ 1805.686469] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:49:19 (1786481359) [ 1806.101108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.115397] alg: No test for adler32 (adler32-zlib) [ 1807.041537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.102665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1807.131513] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1807.776884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1807.857756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg159-client@tcp [ 1814.742634] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:49:28 (1786481368) [ 1815.081674] LNet: 111479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.087798] LNetError: 111479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.106833] LNet: Removed LNI 192.168.201.59@tcp [ 1817.056415] Key type .llcrypt unregistered [ 1817.060825] Key type ._llcrypt unregistered [ 1822.724034] Key type ._llcrypt registered [ 1822.733606] Key type .llcrypt registered [ 1822.925965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1822.944474] alg: No test for adler32 (adler32-zlib) [ 1823.871440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1823.891483] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1823.924431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1830.514876] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:49:44 (1786481384) [ 1832.019413] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1834.202759] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:49:47 (1786481387) [ 1835.681460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1837.078927] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:49:51 (1786481391) [ 1837.496380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1837.520167] LNet: 112904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1837.527959] LNetError: 112904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1837.540189] LNet: Removed LNI 192.168.201.59@tcp [ 1837.554222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1837.589500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1837.607722] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1838.303144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1838.329788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1844.819531] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:49:58 (1786481398) [ 1844.952404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.990691] LNet: 113608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.994668] LNetError: 113608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1845.025330] LNet: Removed LNI 192.168.201.59@tcp [ 1845.064471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.129341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1845.178782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1845.741745] Key type .llcrypt unregistered [ 1845.744842] Key type ._llcrypt unregistered [ 1857.219580] Key type ._llcrypt registered [ 1857.221359] Key type .llcrypt registered [ 1858.785636] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:50:12 (1786481412) [ 1859.504947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1859.515357] alg: No test for adler32 (adler32-zlib) [ 1860.468174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.572563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1860.617696] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1861.423256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1861.473307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1868.653171] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:50:22 (1786481422) [ 1869.060508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.132318] LNet: 115173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1869.148637] LNetError: 115173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.181874] LNet: Removed LNI 192.168.201.59@tcp [ 1869.257770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.380988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1869.444876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1876.703993] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 16:50:30 (1786481430) [ 1876.844678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.873902] LNet: 115753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.887456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.975149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1877.526692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1877.569578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1877.622693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1878.060902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1878.121672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1878.224923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1878.240291] LNet: 115787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.258634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.314389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1878.681494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1878.718318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1878.768910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1879.171874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1879.208905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1885.060493] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 16:50:39 (1786481439) [ 1885.523774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1885.544748] LNet: 116387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.562564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1885.618620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1885.640174] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1886.448881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1886.477907] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1887.655512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1887.713970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1887.933809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1887.986709] LNetError: 116666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.002109] LNet: Removed LNI 192.168.201.59@tcp [ 1888.009980] LNet: Removed LNI 192.168.201.59@tcp2 [ 1888.049863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.127782] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1888.144382] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1888.154938] LNet: 116672:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1888.899294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1889.085345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.101937] LNetError: 116710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.105892] LNetError: 116710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1889.110567] LNet: Removed LNI 192.168.201.59@tcp [ 1889.120896] LNet: Removed LNI 192.168.201.59@tcp2 [ 1889.138668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.184675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1889.205618] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1889.866742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1889.892668] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1891.042681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1891.086839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1891.151589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1891.383637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1891.398404] LNet: 116994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1891.402822] LNet: 116994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1891.407933] LNetError: 116994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1891.413822] LNetError: 116994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1891.419263] LNet: Removed LNI 192.168.201.59@tcp [ 1892.453709] LNet: Removed LNI 192.168.201.59@tcp2 [ 1892.477614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1892.550574] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1892.557702] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1892.566385] LNet: 117000:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1893.259778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1893.547405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.567874] LNet: Removed LNI 192.168.201.59@tcp [ 1894.638734] LNet: Removed LNI 192.168.201.59@tcp2 [ 1894.721740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.791679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1894.840482] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1895.773984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1895.807428] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1896.669397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1896.718082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1896.927987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.948266] LNetError: 117317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.954731] LNetError: 117317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1898.015348] LNet: Removed LNI 192.168.201.59@tcp [ 1898.022897] LNet: Removed LNI 192.168.201.59@tcp2 [ 1898.076393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.229496] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1898.246173] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1898.260778] LNet: 117323:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1898.943070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1899.225359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.255431] LNet: Removed LNI 192.168.201.59@tcp [ 1900.267987] LNet: Removed LNI 192.168.201.59@tcp2 [ 1900.312623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.374784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1900.411920] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1901.178801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1901.202833] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1902.165487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1902.205919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1902.273617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1902.541359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1902.563139] LNet: 117645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1902.566973] LNet: 117645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1903.586300] LNet: Removed LNI 192.168.201.59@tcp [ 1904.612822] LNet: Removed LNI 192.168.201.59@tcp2 [ 1904.642283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.705340] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1904.710470] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1904.715682] LNet: 117652:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1905.251405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1905.502507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.532806] LNetError: 117689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.541979] LNetError: 117689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1905.552281] LNet: Removed LNI 192.168.201.59@tcp [ 1906.603930] LNet: Removed LNI 192.168.201.59@tcp2 [ 1906.655758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.754257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1906.778532] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1907.454534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1907.487799] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1908.325862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1908.356718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1908.406208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1908.783370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.855252] LNet: Removed LNI 192.168.201.59@tcp [ 1909.889887] LNet: Removed LNI 192.168.201.59@tcp2 [ 1909.934199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.089727] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1910.096331] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1910.101185] LNet: 117992:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1910.848488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1911.203974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.223243] LNet: Removed LNI 192.168.201.59@tcp [ 1912.236351] LNet: Removed LNI 192.168.201.59@tcp2 [ 1912.275857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.339908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1912.375317] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1912.999506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1913.020280] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1913.758985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1913.790829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1913.825394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1913.870758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1914.160407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.231150] LNet: Removed LNI 192.168.201.59@tcp [ 1916.275319] LNet: Removed LNI 192.168.201.59@tcp2 [ 1916.317337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.394414] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1916.404703] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1916.967424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1923.423786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1925.136981] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 16:51:18 (1786481478) [ 1925.272224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1925.303305] LNet: 119106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.312208] LNet: 119106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1925.328208] LNetError: 119106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.335184] LNetError: 119106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1925.345984] LNet: Removed LNI 192.168.201.59@tcp [ 1926.380761] LNet: Removed LNI 192.168.201.59@tcp2 [ 1926.425914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.790526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1926.820565] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1927.629593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1927.652282] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1928.819272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.849506] LNet: Removed LNI 10.1.2.3@tcp [ 1928.867283] LNet: Removed LNI 192.168.201.59@tcp [ 1928.887952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.960778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1929.020419] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1929.029963] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1929.759895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1929.793504] LNet: Removed LNI 192.168.201.59@tcp [ 1929.820792] LNet: Removed LNI 10.1.2.3@tcp [ 1936.336862] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 16:51:30 (1786481490) [ 1936.659979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.715940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.763508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1936.791043] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1937.550202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1937.591919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1937.636995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1937.672950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1937.706603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1937.754454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1937.790936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1937.815789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1937.853090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1937.887863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1937.920689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1938.107419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1944.966415] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:51:38 (1786481498) [ 1945.061838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.152105] LNet: Removed LNI 192.168.201.59@tcp [ 1946.176667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.364506] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1946.622820] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1946.631887] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1946.648054] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1949.080858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1949.111636] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1949.811883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1949.846892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1951.992981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1952.030335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1952.080476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.59@tcp -m PUT -e local_timeout -r 1 [ 1952.135781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1952.208346] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1952.212244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1952.398105] LNetError: 120791:0:(lib-move.c:4801:lnet_parse()) 192.168.201.59@tcp, src 192.168.201.59@tcp: Dropping PUT (error -108 looking up sender) [ 1953.440664] LNet: Removed LNI 10.1.2.3@tcp [ 1954.469847] LNet: Removed LNI 192.168.201.59@tcp [ 1954.990959] Key type .llcrypt unregistered [ 1954.998242] Key type ._llcrypt unregistered [ 1968.550684] Key type ._llcrypt registered [ 1968.560252] Key type .llcrypt registered [ 1970.106081] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:52:03 (1786481523) [ 1970.811220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1970.826956] alg: No test for adler32 (adler32-zlib) [ 1971.792962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.278941] LNet: 122108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1973.004373] Key type .llcrypt unregistered [ 1973.007609] Key type ._llcrypt unregistered [ 1979.649762] Key type ._llcrypt registered [ 1979.655259] Key type .llcrypt registered [ 1979.885328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1979.919155] alg: No test for adler32 (adler32-zlib) [ 1980.857881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1980.919117] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1988.742792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1988.972976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1989.606954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.159@tcp [ 1989.828945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1989.852245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1989.873470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1989.900800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1989.926550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1989.970141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1990.011785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1990.044425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1990.066190] LNet: 122561:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1990.083567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1990.119239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1990.160763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1990.175587] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 1990.188136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1990.240773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1990.307040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1990.802175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1991.020849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1991.053036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1991.099136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1991.143543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1991.174634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1991.205788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1991.244611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1991.280928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1991.299778] LNet: 122679:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1991.326697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1991.349756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1991.393617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1991.434578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1991.475671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1991.522171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1992.039857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.159@tcp [ 1992.258823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1992.290387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1992.326220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1992.358797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1992.386374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1992.426139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1992.458371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1992.495585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1992.529577] LNet: 122796:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1992.581811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1992.624882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1992.665614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1992.690950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1992.708840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1992.746495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1993.231987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1993.444728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1993.468987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1993.489821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1993.519957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1993.547823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1993.581795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1993.621773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1993.644622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1993.679771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1993.719911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1993.767731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1993.812808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1993.834985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1993.857275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1994.183633] LNet: 122963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1994.193272] LNetError: 122963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1994.210755] LNet: Removed LNI 192.168.201.59@tcp [ 1994.497676] Key type .llcrypt unregistered [ 1994.499101] Key type ._llcrypt unregistered [ 2010.345723] Key type ._llcrypt registered [ 2010.347303] Key type .llcrypt registered [ 2011.553159] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:52:45 (1786481565) [ 2011.911157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.925263] alg: No test for adler32 (adler32-zlib) [ 2012.851859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.272216] LNet: 123851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.715494] Key type .llcrypt unregistered [ 2013.717578] Key type ._llcrypt unregistered [ 2020.034057] Key type ._llcrypt registered [ 2020.036196] Key type .llcrypt registered [ 2020.205579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2020.219256] alg: No test for adler32 (adler32-zlib) [ 2021.119184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2021.140972] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2031.026941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2041.198400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.260331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2041.303887] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 2042.310835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2043.074603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.159@tcp [ 2043.331266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2043.394877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2043.444887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2043.504910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2043.540196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2043.554904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2043.567911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2043.586221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2043.595954] LNet: 124507:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2043.607687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2043.630660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2043.657907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2043.679296] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 2043.694131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2043.718457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2043.743906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2044.498623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2044.846144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2044.926567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2045.021360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2045.151723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2045.227849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2045.268579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2045.338424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2045.392629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2045.436697] LNet: 124625:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2045.468560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2045.532761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2045.575277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2045.625161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2045.686292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2045.741121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2046.431382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.159@tcp [ 2046.636861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2046.666224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2046.694445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2046.719790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2046.748898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2046.778680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2046.811218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2046.842692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2046.866917] LNet: 124742:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2046.888338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2046.931217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2046.964418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2046.996447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2047.027536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2047.060841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2047.843548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2048.173839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2048.215121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2048.265237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2048.308396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2048.353759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2048.390965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2048.423458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2048.462346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2048.515763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2048.569703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2048.603485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2048.651458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2048.688813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2048.741969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2055.529895] LNet: 124944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.534934] LNetError: 124944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.541560] LNet: Removed LNI 192.168.201.59@tcp [ 2055.557598] LNet: Removed LNI 192.168.201.59@tcp1 [ 2056.041335] Key type .llcrypt unregistered [ 2056.048780] Key type ._llcrypt unregistered [ 2074.164308] Key type ._llcrypt registered [ 2074.166712] Key type .llcrypt registered [ 2075.479324] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 16:53:49 (1786481629) [ 2075.944435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.953935] alg: No test for adler32 (adler32-zlib) [ 2076.922725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.029979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2077.066916] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2077.921544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2077.940980] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2078.917642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2079.017739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2080.095169] LNet: Removed LNI 192.168.201.59@tcp [ 2087.297333] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 16:54:00 (1786481640) [ 2089.060088] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2090.908743] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 16:54:04 (1786481644) [ 2099.443582] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 16:54:13 (1786481653) [ 2107.320509] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 16:54:21 (1786481661) [ 2115.635777] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 16:54:29 (1786481669) [ 2122.433649] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 16:54:36 (1786481676) [ 2129.886944] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 16:54:43 (1786481683) [ 2137.176863] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 16:54:50 (1786481690) [ 2144.365549] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 16:54:58 (1786481698) [ 2150.450931] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 16:55:04 (1786481704) [ 2157.536800] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 16:55:11 (1786481711) [ 2164.020329] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 16:55:17 (1786481717) [ 2172.502539] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 16:55:26 (1786481726) [ 2179.212320] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:55:33 (1786481733) [ 2186.676337] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:55:40 (1786481740) [ 2194.682146] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:55:48 (1786481748) [ 2202.909563] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:55:56 (1786481756) [ 2210.915547] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:56:04 (1786481764) [ 2218.226814] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:56:11 (1786481771) [ 2218.389774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2218.442281] LNet: 136632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.449670] LNetError: 136632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.482880] LNet: Removed LNI 192.168.201.59@tcp2 [ 2218.533938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.711187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2218.745114] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2219.918418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.935516] LNet: 136779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.940773] LNetError: 136779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.963241] LNet: Removed LNI 192.168.201.59@tcp [ 2219.982753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.045515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2220.083954] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2227.496132] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:56:21 (1786481781) [ 2235.423177] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:56:29 (1786481789) [ 2242.981785] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:56:36 (1786481796) [ 2250.006523] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:56:43 (1786481803) [ 2256.653536] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:56:50 (1786481810) [ 2263.249173] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:56:57 (1786481817) [ 2270.672743] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:57:04 (1786481824) [ 2278.239086] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:57:12 (1786481832) [ 2284.594786] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:57:18 (1786481838) [ 2291.846662] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:57:25 (1786481845) [ 2298.457575] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:57:32 (1786481852) [ 2305.930635] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:57:39 (1786481859) [ 2311.991384] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2187 sec ========== 16:57:45 (1786481865) [ 2312.340209] LNet: 144243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.344147] LNetError: 144243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.369991] LNet: Removed LNI 192.168.201.59@tcp [ 2312.967652] Key type .llcrypt unregistered [ 2312.969788] Key type ._llcrypt unregistered [ 2323.795686] Key type ._llcrypt registered [ 2323.797564] Key type .llcrypt registered [ 2324.071711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2324.091277] alg: No test for adler32 (adler32-zlib) [ 2325.149908] Lustre: Lustre: Build Version: 2.17.56_51_g4226bd4 [ 2325.370175] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2327.039281] Key type lgssc registered [ 2327.985380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2377.002824] Lustre: Mounted lustre-client - version 2.17.56_51_g4226bd4 [ 2381.031415] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2386.393308] Lustre: Unmounted lustre-client [ 2419.516944] Key type lgssc unregistered [ 2419.761775] LNet: 147200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.772166] LNetError: 147200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.785841] LNet: Removed LNI 192.168.201.59@tcp [ 2420.343213] Key type .llcrypt unregistered [ 2420.345241] Key type ._llcrypt unregistered