[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 428745157 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003295] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009005] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010025] pid_max: default: 32768 minimum: 301 [ 0.012135] LSM: Security Framework initializing [ 0.013058] Yama: becoming mindful. [ 0.014028] SELinux: Initializing. [ 0.015056] *** VALIDATE selinux *** [ 0.023807] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027429] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028118] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029083] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031080] *** VALIDATE tmpfs *** [ 0.032425] *** VALIDATE proc *** [ 0.033245] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.036213] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043161] debug: unmapping init [mem 0xffffffffaac59000-0xffffffffaac60fff] [ 0.045976] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047034] ... version: 2 [ 0.048011] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053016] ... event mask: 000000070000000f [ 0.054277] rcu: Hierarchical SRCU implementation. [ 0.056369] smp: Bringing up secondary CPUs ... [ 0.057477] x86: Booting SMP configuration: [ 0.058020] .... node #0, CPUs: #1 #2 #3 [ 0.061015] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.110511] node 0 deferred pages initialised in 44ms [ 0.112094] devtmpfs: initialized [ 0.113225] x86/mm: Memory block size: 128MB [ 0.115472] gcov: version magic: 0x41383552 [ 0.117035] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.118076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.119239] pinctrl core: initialized pinctrl subsystem [ 0.120193] [ 0.120778] ************************************************************* [ 0.121010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.122012] ** ** [ 0.123008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.124007] ** ** [ 0.125008] ** This means that this kernel is built to expose internal ** [ 0.126009] ** IOMMU data structures, which may compromise security on ** [ 0.127012] ** your system. ** [ 0.128010] ** ** [ 0.129007] ** If you see this message and you are not debugging the ** [ 0.130009] ** kernel, report this immediately to your vendor! ** [ 0.131022] ** ** [ 0.132011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.133010] ************************************************************* [ 0.134707] NET: Registered protocol family 16 [ 0.135415] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.136079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.137045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.138395] cpuidle: using governor menu [ 0.140794] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.143452] PCI: Using configuration type 1 for base access [ 0.146145] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.158060] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.159029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.160163] cryptd: max_cpu_qlen set to 1000 [ 0.162193] ACPI: Added _OSI(Module Device) [ 0.163016] ACPI: Added _OSI(Processor Device) [ 0.164024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.165011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.169581] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.171481] ACPI: Interpreter enabled [ 0.172065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.173011] ACPI: Using IOAPIC for interrupt routing [ 0.174097] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.175369] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.184622] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.185041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.186028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.187084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.189266] acpiphp: Slot [2] registered [ 0.190147] acpiphp: Slot [5] registered [ 0.191146] acpiphp: Slot [6] registered [ 0.192228] acpiphp: Slot [3] registered [ 0.194105] acpiphp: Slot [4] registered [ 0.196105] acpiphp: Slot [7] registered [ 0.197091] acpiphp: Slot [8] registered [ 0.199106] acpiphp: Slot [9] registered [ 0.200123] acpiphp: Slot [10] registered [ 0.202162] acpiphp: Slot [11] registered [ 0.204100] acpiphp: Slot [12] registered [ 0.205111] acpiphp: Slot [13] registered [ 0.207109] acpiphp: Slot [14] registered [ 0.209102] acpiphp: Slot [15] registered [ 0.210132] acpiphp: Slot [16] registered [ 0.212117] acpiphp: Slot [17] registered [ 0.213113] acpiphp: Slot [18] registered [ 0.215106] acpiphp: Slot [19] registered [ 0.217113] acpiphp: Slot [20] registered [ 0.218129] acpiphp: Slot [21] registered [ 0.220103] acpiphp: Slot [22] registered [ 0.221150] acpiphp: Slot [23] registered [ 0.223115] acpiphp: Slot [24] registered [ 0.225098] acpiphp: Slot [25] registered [ 0.226105] acpiphp: Slot [26] registered [ 0.228110] acpiphp: Slot [27] registered [ 0.230103] acpiphp: Slot [28] registered [ 0.231135] acpiphp: Slot [29] registered [ 0.233099] acpiphp: Slot [30] registered [ 0.235103] acpiphp: Slot [31] registered [ 0.236101] PCI host bridge to bus 0000:00 [ 0.238023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.240023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.243025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.246032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.248051] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.251071] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.253344] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.255987] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.259424] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.266937] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.270990] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.272022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.275024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.277019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.279597] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.281668] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.284063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.288135] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.293025] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.303020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.307651] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.314898] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.320012] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.324018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.338018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.346942] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.355019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.364015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.378013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.387416] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.390405] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.392372] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.394339] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.397297] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.401193] iommu: Default domain type: Passthrough [ 0.404334] SCSI subsystem initialized [ 0.405000] ACPI: bus type USB registered [ 0.408127] usbcore: registered new interface driver usbfs [ 0.410082] usbcore: registered new interface driver hub [ 0.412087] usbcore: registered new device driver usb [ 0.414187] pps_core: LinuxPPS API ver. 1 registered [ 0.416012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.419058] PTP clock support registered [ 0.421144] EDAC MC: Ver: 3.0.0 [ 0.424123] PCI: Using ACPI for IRQ routing [ 0.425594] NetLabel: Initializing [ 0.427015] NetLabel: domain hash size = 128 [ 0.429011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.431085] NetLabel: unlabeled traffic allowed by default [ 0.433121] vgaarb: loaded [ 0.435249] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.437014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.443369] clocksource: Switched to clocksource kvm-clock [ 0.549705] VFS: Disk quotas dquot_6.6.0 [ 0.551587] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.554297] *** VALIDATE ramfs *** [ 0.555563] *** VALIDATE hugetlbfs *** [ 0.557146] pnp: PnP ACPI init [ 0.559526] pnp: PnP ACPI: found 6 devices [ 0.578917] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.582487] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.584675] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.587739] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.590523] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.593835] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.596909] NET: Registered protocol family 2 [ 0.598840] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.603561] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.606652] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.611638] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.614743] TCP: Hash tables configured (established 65536 bind 65536) [ 0.617425] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.619964] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.622424] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.624853] NET: Registered protocol family 1 [ 0.627726] RPC: Registered named UNIX socket transport module. [ 0.629734] RPC: Registered udp transport module. [ 0.630775] RPC: Registered tcp transport module. [ 0.632099] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.634215] NET: Registered protocol family 44 [ 0.636015] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.637970] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.639853] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.641878] PCI: CLS 0 bytes, default 64 [ 0.643872] Unpacking initramfs... [ 2.166746] debug: unmapping init [mem 0xffff9530fcc64000-0xffff9530fffcffff] [ 2.172422] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.175011] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.178446] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.679239] Initialise system trusted keyrings [ 2.681215] Key type blacklist registered [ 2.683306] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.691711] zbud: loaded [ 2.694836] *** VALIDATE nfs *** [ 2.695932] *** VALIDATE nfs4 *** [ 2.697851] pstore: using deflate compression [ 2.701484] Platform Keyring initialized [ 2.808269] NET: Registered protocol family 38 [ 2.809916] Key type asymmetric registered [ 2.811495] Asymmetric key parser 'x509' registered [ 2.812788] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.814979] io scheduler mq-deadline registered [ 2.816224] io scheduler kyber registered [ 2.817489] io scheduler bfq registered [ 2.819093] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.821426] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.823420] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.825284] ACPI: Power Button [PWRF] [ 2.830586] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.835892] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.847876] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.875836] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.906114] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.910300] Non-volatile memory driver v1.3 [ 2.911617] Linux agpgart interface v0.103 [ 2.938618] virtio_blk virtio1: [vda] 136576 512-byte logical blocks (69.9 MB/66.7 MiB) [ 2.941687] vda: detected capacity change from 0 to 69926912 [ 2.955558] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.958875] vdb: detected capacity change from 0 to 1073741824 [ 2.966799] libphy: Fixed MDIO Bus: probed [ 2.971617] usbcore: registered new interface driver usbserial_generic [ 2.974312] usbserial: USB Serial support registered for generic [ 2.977131] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.981942] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.983898] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.986521] mousedev: PS/2 mouse device common for all mice [ 2.989474] rtc_cmos 00:05: RTC can wake from S4 [ 2.992209] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.996635] rtc_cmos 00:05: registered as rtc0 [ 2.998276] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.001900] intel_pstate: CPU model not supported [ 3.004937] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.009222] hid: raw HID events driver (C) Jiri Kosina [ 3.011686] usbcore: registered new interface driver usbhid [ 3.013819] usbhid: USB HID core driver [ 3.014071] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.015163] drop_monitor: Initializing network drop monitor service [ 3.021844] Initializing XFRM netlink socket [ 3.024202] NET: Registered protocol family 10 [ 3.027780] Segment Routing with IPv6 [ 3.029629] NET: Registered protocol family 17 [ 3.032092] mpls_gso: MPLS GSO support [ 3.038073] RAS: Correctable Errors collector initialized. [ 3.041133] AVX version of gcm_enc/dec engaged. [ 3.042878] AES CTR mode by8 optimization enabled [ 3.122839] sched_clock: Marking stable (3122817430, 0)->(4028438385, -905620955) [ 3.126946] registered taskstats version 1 [ 3.129397] Loading compiled-in X.509 certificates [ 3.131748] zswap: loaded using pool lzo/zbud [ 3.155807] Key type big_key registered [ 3.167887] Key type encrypted registered [ 3.169671] ima: No TPM chip found, activating TPM-bypass! [ 3.172188] ima: Allocated hash algorithm: sha1 [ 3.173954] ima: No architecture policies found [ 3.175661] evm: Initialising EVM extended attributes: [ 3.177439] evm: security.selinux [ 3.178421] evm: security.ima [ 3.179352] evm: security.capability [ 3.180680] evm: HMAC attrs: 0x1 [ 3.182912] rtc_cmos 00:05: setting system clock to 2026-06-12 04:26:00 UTC (1781238360) [ 3.189443] debug: unmapping init [mem 0xffffffffabc03000-0xffffffffabdfffff] [ 3.191989] debug: unmapping init [mem 0xffffffffaa982000-0xffffffffaac58fff] [ 3.201602] Write protecting the kernel read-only data: 28672k [ 3.205090] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 3.207906] debug: unmapping init [mem 0xffffffffa9914000-0xffffffffa99fffff] [ 3.238389] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.247053] systemd[1]: Detected virtualization kvm. [ 3.249035] systemd[1]: Detected architecture x86-64. [ 3.250847] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.275138] systemd[1]: No hostname configured. [ 3.276993] systemd[1]: Set hostname to . [ 3.279189] random: systemd: uninitialized urandom read (16 bytes read) [ 3.282079] systemd[1]: Initializing machine ID from random generator. [ 3.328602] random: ln: uninitialized urandom read (6 bytes read) [ 3.408167] random: systemd: uninitialized urandom read (16 bytes read) [ 3.410748] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.415588] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.425245] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.040200] device-mapper: uevent: version 1.0.3 [ 4.042105] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.740800] virtio_net virtio0 ens2: renamed from eth0 [ 4.772992] random: fast init done [ 4.773208] scsi host0: ata_piix [ 4.820366] scsi host1: ata_piix [ 4.821981] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.826661] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.580312] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.685753] random: crng init done [ 9.687438] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.107804] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.220870] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.479983] SELinux: Disabled at runtime. [ 11.550884] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.560349] systemd[1]: Detected virtualization kvm. [ 11.562067] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.027186] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.032802] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.044540] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.047878] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.050157] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.057133] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.060718] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ 12.101053] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.465604] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.728349] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.772673] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.884852] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.898153] EDAC sbridge: Ver: 1.1.2 [ 13.901617] Key type dns_resolver registered [ 14.197704] NFS: Registering the id_resolver key type [ 14.199944] Key type id_resolver registered [ 14.201576] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg305-client login: [ 29.167070] hrtimer: interrupt took 11062064 ns [ 63.872225] libcfs: loading out-of-tree module taints kernel. [ 64.187580] Key type ._llcrypt registered [ 64.201154] Key type .llcrypt registered [ 64.666166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 64.676887] alg: No test for adler32 (adler32-zlib) [ 66.328836] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 67.338136] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 69.152663] Key type lgssc registered [ 71.045111] Lustre: Echo OBD driver; http://www.lustre.org/ [ 192.546301] Lustre: Mounted lustre-client [ 197.105628] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 208.856314] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing check_logdir /tmp/testlogs/ [ 214.557331] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing yml_node [ 218.081513] Lustre: lustre-OST0000-osc-ffff953144c90000: disconnect after 23s idle [ 220.853794] Lustre: DEBUG MARKER: Client: 2.17.53.78 [ 224.130775] Lustre: DEBUG MARKER: MDS: 2.17.53.78 [ 226.835909] Lustre: DEBUG MARKER: OSS: 2.17.53.78 [ 228.968864] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jun 12 00:29:44 EDT 2026 [ 246.113122] Lustre: Unmounted lustre-client [ 281.752890] Key type lgssc unregistered [ 282.069789] LNet: 8384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.075302] LNetError: 8384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 282.091846] LNet: Removed LNI 192.168.203.5@tcp [ 282.923157] Key type .llcrypt unregistered [ 282.929115] Key type ._llcrypt unregistered [ 307.770416] Key type ._llcrypt registered [ 307.773045] Key type .llcrypt registered [ 307.919061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 307.941628] alg: No test for adler32 (adler32-zlib) [ 308.908244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 309.033596] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 309.056461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 313.319898] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 313.675703] LNet: 9268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.684114] LNetError: 9268:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 313.712349] LNet: Removed LNI 192.168.203.5@tcp [ 314.416413] Key type .llcrypt unregistered [ 314.419453] Key type ._llcrypt unregistered [ 321.000645] Key type ._llcrypt registered [ 321.006083] Key type .llcrypt registered [ 322.706310] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:31:18 (1781238678) [ 323.041619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 323.066293] alg: No test for adler32 (adler32-zlib) [ 324.062864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.152267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.192336] LNet: 9679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.209065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.287552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 324.304467] LNet: 9686:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 331.885975] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:31:27 (1781238687) [ 331.980745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.013922] LNet: 10257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.036563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.199522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 338.566953] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:31:34 (1781238694) [ 338.749422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.780431] LNet: 10851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.800904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.876409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 345.128975] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:31:41 (1781238701) [ 345.279730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.302597] LNet: 11436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.328492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.399937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 351.428496] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:31:47 (1781238707) [ 351.714250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.754660] LNet: 12022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.797745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.875982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 352.468987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 360.069290] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:31:56 (1781238716) [ 360.173166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.208215] LNet: 12607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.239587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.307553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 360.727623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 361.182823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 367.677779] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:32:03 (1781238723) [ 367.841844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.923924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.000258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 374.718381] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:32:10 (1781238730) [ 374.894903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.938757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.002431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 375.033459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 375.412978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 375.450913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 375.937081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 375.981259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 376.391377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 376.420231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 376.914311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 376.958773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 377.428226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 377.476566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 378.048480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 378.100662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 378.405934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 378.459301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 378.910548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 378.951396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 379.312849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 379.361321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 379.826842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 379.856899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 380.241088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 380.275941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 380.643112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 380.674713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 386.832715] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:32:22 (1781238742) [ 386.912635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.941140] LNet: 14470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.951909] LNet: 14470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 386.979397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.073825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 387.106313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 393.686101] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:32:29 (1781238749) [ 393.832417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.888545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.984524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 394.029474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 401.168926] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:32:37 (1781238757) [ 401.326962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.420730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.503713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 401.536788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 408.258450] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:32:44 (1781238764) [ 408.352265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.420866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.547821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 408.583271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 415.403994] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:32:51 (1781238771) [ 415.525334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.573823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.641958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 415.696563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 422.034553] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:32:58 (1781238778) [ 422.338304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.364874] LNet: 17407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.370657] LNet: 17407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 422.395626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.487120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 422.525520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 429.180887] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:33:05 (1781238785) [ 429.243452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.300529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.435609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 429.769519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 435.925183] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:33:12 (1781238792) [ 436.139440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.196498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.416981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.603751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 443.580818] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:33:19 (1781238799) [ 443.717883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.817231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.026482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 444.285686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 450.812795] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:33:26 (1781238806) [ 450.952967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.011993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.560886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 451.888365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 458.368366] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:33:34 (1781238814) [ 458.497322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.557689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.703678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.754680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.026952] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:33:40 (1781238820) [ 465.116377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.175562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.274548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.343611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.023691] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:33:47 (1781238827) [ 472.169270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.271460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.396917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 478.942446] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:33:55 (1781238835) [ 479.237261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.303397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.435880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 486.720035] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:34:02 (1781238842) [ 486.854144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.893989] LNet: 22840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 486.904089] LNet: 22840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 486.933334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.035032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 487.610342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 493.751399] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:34:09 (1781238849) [ 493.861162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.920688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.052665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 494.084614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 500.797295] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:34:16 (1781238856) [ 500.881606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.923561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.022171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 501.063920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 506.702617] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:34:22 (1781238862) [ 506.894892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.929970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.015290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 507.043758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 512.623368] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:34:28 (1781238868) [ 512.832263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.875641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.954353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 512.995494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 519.302354] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:34:35 (1781238875) [ 519.502195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.535843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.581474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 519.662776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 519.680823] LNetError: 25791:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 519.730401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 519.774450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 527.598503] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:34:42 (1781238882) [ 527.717411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.782467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.924837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lYKsxcBlmz [ 528.026934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lYKsxcBlmz [ 528.109922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lYKsxcBlmz [ 535.186114] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:34:50 (1781238890) [ 535.289771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.385795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.548464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 535.613304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 535.690873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 543.605203] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:34:59 (1781238899) [ 543.838916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.859480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.904610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 546.175624] LNet: Message forwarding will be enabled in 2s [ 549.282600] LNet: Message forwarding enabled [ 549.606763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 549.629879] LNet: Message forwarding will stop in 2s to 6s [ 554.187694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 554.625450] LNet: Message forwarding disabled [ 555.783517] LNet: Message forwarding will be enabled in 2s [ 559.777073] LNet: Message forwarding enabled [ 562.692863] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:35:18 (1781238918) [ 563.135245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.287501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.415612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 570.985861] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:35:26 (1781238926) [ 571.110918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.155856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.231442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 571.289638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 571.350552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 571.399902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 571.447494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 571.478932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 571.507584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 571.539636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 571.562503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 571.598660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 571.640186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 571.686458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 571.722572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 571.748470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 571.801441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 571.856199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 571.890588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 571.917580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 571.959219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 571.987691] LNetError: 28863:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 578.048725] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:35:34 (1781238934) [ 578.320479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.355239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.435938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 584.947515] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:35:41 (1781238941) [ 587.036549] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg305-server [ 588.572951] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:35:44 (1781238944) [ 588.860482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.912815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.990213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 589.016304] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 590.784643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[6-10]@tcp [ 590.802910] LNet: 30579:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 590.848518] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.7@tcp is being used as a gateway but routing feature is not turned on [ 590.863812] LNetError: 30368:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.7@tcp has gone from up to down [ 591.846087] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 591.858921] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 592.871441] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.9@tcp is being used as a gateway but routing feature is not turned on [ 592.885608] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 594.916885] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 594.939824] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 597.663592] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:35:53 (1781238953) [ 599.074900] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.9@tcp is being used as a gateway but routing feature is not turned on [ 599.087508] LNetError: 30368:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 599.290486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg305-server [ 600.908943] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:35:56 (1781238956) [ 601.003376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.039578] LNetError: 31510:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.074710] LNet: Removed LNI 192.168.203.5@tcp [ 601.120384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.240103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 601.279333] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 601.958460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[6-11/2]@tcp [ 601.982732] LNet: 31631:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 601.988797] LNet: 31631:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 602.004959] LNetError: 31524:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.10@tcp has gone from up to down [ 602.010406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[6-11/2]@tcp [ 602.013395] LNetError: 31524:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 608.215617] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:36:04 (1781238964) [ 608.423609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.451526] LNetError: 32223:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.466770] LNet: Removed LNI 192.168.203.5@tcp [ 608.482052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.571248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 608.644018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.687385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.747384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 608.805499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.855718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.906559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 608.976134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.011789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.071235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.148315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.182950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.236833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 609.321976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.360764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.438958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.527273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.574680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.643347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 609.745118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.779127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.846363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.970456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.005853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.064667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 610.145880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.191501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.283443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 610.377985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.436592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.505409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 610.593121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.631245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.758597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 617.089146] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:36:13 (1781238973) [ 619.407701] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg305-server [ 621.184401] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:36:17 (1781238977) [ 622.904239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg305-server [ 624.492266] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:36:20 (1781238980) [ 624.627140] LNet: 33620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.634532] LNet: 33620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 25 previous similar messages [ 625.324966] Key type .llcrypt unregistered [ 625.327374] Key type ._llcrypt unregistered [ 630.338649] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 630.458646] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 630.468540] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 630.477220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 632.873414] Key type ._llcrypt registered [ 632.877544] Key type .llcrypt registered [ 632.999111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 633.007164] alg: No test for adler32 (adler32-zlib) [ 633.882358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.923516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 633.940890] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 634.509359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 634.528530] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 636.051748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 636.069832] LNet: Removed LNI 10.1.2.3@tcp [ 636.179908] LNet: 34339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.188922] LNetError: 34339:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.203745] LNet: Removed LNI 192.168.203.5@tcp [ 636.853982] Key type .llcrypt unregistered [ 636.855838] Key type ._llcrypt unregistered [ 646.055816] Key type ._llcrypt registered [ 646.058348] Key type .llcrypt registered [ 647.313296] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:36:43 (1781239003) [ 647.774061] Key type .llcrypt unregistered [ 647.777736] Key type ._llcrypt unregistered [ 653.847706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 653.957851] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 653.962249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 653.968352] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 656.488233] Key type ._llcrypt registered [ 656.492530] Key type .llcrypt registered [ 656.656616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 656.673234] alg: No test for adler32 (adler32-zlib) [ 657.574629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.633548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 657.663209] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 658.246929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 658.268165] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 660.246498] LNet: 35832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.252959] LNetError: 35832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.257870] LNet: Removed LNI 192.168.203.5@tcp [ 660.265078] LNet: Removed LNI 10.1.2.3@tcp [ 660.853598] Key type .llcrypt unregistered [ 660.856182] Key type ._llcrypt unregistered [ 670.985931] Key type ._llcrypt registered [ 670.987399] Key type .llcrypt registered [ 672.373792] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:37:08 (1781239028) [ 672.970486] Key type .llcrypt unregistered [ 672.974514] Key type ._llcrypt unregistered [ 678.145263] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 678.227659] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 678.231632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 678.235468] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 680.895411] Key type ._llcrypt registered [ 680.901286] Key type .llcrypt registered [ 681.068699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 681.081720] alg: No test for adler32 (adler32-zlib) [ 681.984817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.025547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 682.040532] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 683.972730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 683.994417] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 686.036425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 686.050083] LNet: Removed LNI 10.1.2.3@tcp [ 686.063450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 686.099532] LNet: Removed LNI 10.1.2.31@tcp [ 686.118981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 686.136184] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 686.790256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 686.806329] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 687.606484] LNet: 37768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.610648] LNetError: 37768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 687.615217] LNet: Removed LNI 10.1.2.3@tcp [ 687.630219] LNet: Removed LNI 10.1.2.31@tcp [ 688.082494] Key type .llcrypt unregistered [ 688.084927] Key type ._llcrypt unregistered [ 699.471787] Key type ._llcrypt registered [ 699.477682] Key type .llcrypt registered [ 700.835974] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:37:36 (1781239056) [ 701.307485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 701.325524] alg: No test for adler32 (adler32-zlib) [ 702.286961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.351716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.383551] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 703.236465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.264329] LNet: 38721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.271242] LNetError: 38721:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.282919] LNet: Removed LNI 192.168.203.5@tcp [ 703.296782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.357531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.5@tcp [ 703.393788] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 704.250095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.5@tcp [ 704.286050] LNet: Removed LNI 192.168.203.5@tcp [ 704.339817] LNet: 38857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.976397] Key type .llcrypt unregistered [ 704.979432] Key type ._llcrypt unregistered [ 716.589819] Key type ._llcrypt registered [ 716.593407] Key type .llcrypt registered [ 718.134264] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:37:53 (1781239073) [ 720.100479] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg305-server [ 721.964073] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:37:57 (1781239077) [ 722.412412] Key type .llcrypt unregistered [ 722.415365] Key type ._llcrypt unregistered [ 728.543150] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 728.686636] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 728.698656] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 728.725212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 731.288920] Key type ._llcrypt registered [ 731.293652] Key type .llcrypt registered [ 731.514907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 731.533456] alg: No test for adler32 (adler32-zlib) [ 732.455614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.550527] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 733.371458] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 735.361865] LNet: 40694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.368596] LNetError: 40694:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.379415] LNet: Removed LNI 192.168.203.5@tcp1 [ 735.395119] LNet: Removed LNI 10.1.2.3@tcp2 [ 735.960830] Key type .llcrypt unregistered [ 735.963385] Key type ._llcrypt unregistered [ 746.419616] Key type ._llcrypt registered [ 746.422077] Key type .llcrypt registered [ 747.381904] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:38:23 (1781239103) [ 747.730967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 747.743483] alg: No test for adler32 (adler32-zlib) [ 748.628773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.684333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 748.698337] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 749.354515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.384465] LNet: 41663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.394041] LNetError: 41663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.406472] LNet: Removed LNI 192.168.203.5@tcp [ 749.426904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.484962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 749.521535] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 754.967330] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:38:31 (1781239111) [ 755.106378] LNet: 42251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.111979] LNetError: 42251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.140131] LNet: Removed LNI 192.168.203.5@tcp [ 755.679418] Key type .llcrypt unregistered [ 755.682937] Key type ._llcrypt unregistered [ 760.712256] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 760.814132] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 760.818329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 760.830590] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 763.288385] Key type ._llcrypt registered [ 763.292860] Key type .llcrypt registered [ 763.435420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 763.451428] alg: No test for adler32 (adler32-zlib) [ 764.322445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.376330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 764.397972] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 764.985037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 765.005301] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 766.778994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.794640] LNet: 43014:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.803081] LNetError: 43014:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.812292] LNet: Removed LNI 192.168.203.5@tcp [ 766.821864] LNet: Removed LNI 10.1.2.3@tcp [ 766.848438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 766.909252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 766.925058] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 766.928149] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 767.303551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 767.324350] LNet: 43041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.331041] LNetError: 43041:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.337764] LNetError: 43041:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 767.344074] LNet: Removed LNI 192.168.203.5@tcp [ 767.353508] LNet: Removed LNI 10.1.2.3@tcp [ 767.369286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.426456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 767.448721] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 767.452522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 767.815990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 767.831638] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 768.703772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.727942] LNet: 43204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.733984] LNetError: 43204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.739755] LNetError: 43204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 768.744543] LNet: Removed LNI 192.168.203.5@tcp [ 768.746935] LNet: Removed LNI 10.1.2.3@tcp [ 768.759517] LNet: Removed LNI 192.168.203.5@tcp2 [ 768.783344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.839314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 768.862983] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 768.868194] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 768.872702] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 773.786349] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:38:50 (1781239130) [ 773.894531] LNet: 43797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.902185] LNetError: 43797:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.906181] LNetError: 43797:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 773.913325] LNet: Removed LNI 192.168.203.5@tcp [ 773.928210] LNet: Removed LNI 10.1.2.3@tcp [ 773.939651] LNet: Removed LNI 192.168.203.5@tcp2 [ 774.442353] Key type .llcrypt unregistered [ 774.445525] Key type ._llcrypt unregistered [ 778.885824] Key type ._llcrypt registered [ 778.894941] Key type .llcrypt registered [ 779.133850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.157414] alg: No test for adler32 (adler32-zlib) [ 780.048324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.096667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.127363] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 780.785285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.810569] LNet: 44196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.816727] LNetError: 44196:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.839131] LNet: Removed LNI 192.168.203.5@tcp [ 780.850224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.902800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 780.922599] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 781.323610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.344810] LNet: 44226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.350375] LNetError: 44226:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.366754] LNet: Removed LNI 192.168.203.5@tcp [ 781.382125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.426130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 781.455530] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 781.868997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.925215] LNet: Removed LNI 192.168.203.5@tcp [ 781.953842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.045834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 782.066296] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 782.484484] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 782.674205] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 782.679119] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 782.681914] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 785.072545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.100328] LNet: 44336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.106382] LNet: 44336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 785.119810] LNetError: 44336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.134624] LNetError: 44336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 785.150388] LNet: Removed LNI 192.168.203.5@tcp [ 785.187596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.267447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 785.303853] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 790.946655] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:39:07 (1781239147) [ 791.029847] LNet: 44924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.033381] LNetError: 44924:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.045851] LNet: Removed LNI 192.168.203.5@tcp [ 791.611920] Key type .llcrypt unregistered [ 791.613868] Key type ._llcrypt unregistered [ 796.565733] Key type ._llcrypt registered [ 796.567390] Key type .llcrypt registered [ 796.713933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 796.727730] alg: No test for adler32 (adler32-zlib) [ 797.612908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.663755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.686608] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 798.478780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.499555] LNet: 45331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.504756] LNetError: 45331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.523712] LNet: Removed LNI 192.168.203.5@tcp [ 798.554139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.632909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 798.652901] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 804.313507] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:39:20 (1781239160) [ 804.569541] LNet: 45923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.576419] LNetError: 45923:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.588943] LNet: Removed LNI 192.168.203.5@tcp [ 805.202604] Key type .llcrypt unregistered [ 805.204940] Key type ._llcrypt unregistered [ 810.574458] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 810.728272] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 810.732373] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 810.741919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 813.188512] Key type ._llcrypt registered [ 813.190305] Key type .llcrypt registered [ 813.330646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 813.348446] alg: No test for adler32 (adler32-zlib) [ 814.280958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.345404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.369435] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 815.126827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.143285] LNet: 46367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.147580] LNetError: 46367:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.167203] LNet: Removed LNI 192.168.203.5@tcp [ 815.196816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.244863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 815.266135] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 821.422872] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:39:37 (1781239177) [ 821.605208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.620437] LNet: 46967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.626401] LNetError: 46967:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.645045] LNet: Removed LNI 192.168.203.5@tcp [ 821.664157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.774800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 826.620342] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:39:42 (1781239182) [ 826.856839] LNet: 47548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.246573] Key type .llcrypt unregistered [ 827.250072] Key type ._llcrypt unregistered [ 832.509486] Key type ._llcrypt registered [ 832.512377] Key type .llcrypt registered [ 832.715625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 832.729741] alg: No test for adler32 (adler32-zlib) [ 833.588379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.648756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 833.702201] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 834.634519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.649411] LNet: 47946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.653607] LNetError: 47946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.668053] LNet: Removed LNI 192.168.203.5@tcp [ 834.685252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.741499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 834.769069] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 842.267157] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:39:58 (1781239198) [ 842.358085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.371801] LNet: 48532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.377684] LNetError: 48532:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.395323] LNet: Removed LNI 192.168.203.5@tcp [ 842.429682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.528056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 849.019872] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:40:05 (1781239205) [ 851.460547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 853.361317] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:40:09 (1781239209) [ 855.239896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 857.451953] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:40:13 (1781239213) [ 858.724251] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 861.116745] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:40:16 (1781239216) [ 862.881585] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 864.773073] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:40:20 (1781239220) [ 866.357727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 868.483541] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 00:40:24 (1781239224) [ 868.873405] LNet: 50590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.552801] Key type .llcrypt unregistered [ 869.556468] Key type ._llcrypt unregistered [ 876.269647] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 876.427898] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 876.431419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 876.435740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 879.036452] Key type ._llcrypt registered [ 879.042439] Key type .llcrypt registered [ 879.270964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.286995] alg: No test for adler32 (adler32-zlib) [ 880.160799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.215258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 880.230935] LNet: Added LNI 192.168.203.5@tcp [8/512/0/180] [ 880.235569] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 880.238399] LNet: 50911:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 880.315670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.333646] LNet: 50930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.339842] LNetError: 50930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.346210] LNet: Removed LNI 192.168.203.5@tcp [ 880.348882] LNetError: 50930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 880.357838] LNet: Removed LNI 10.1.2.3@tcp [ 880.381295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.465686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 880.512904] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 880.613282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.654900] LNet: 50954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.674985] LNetError: 50954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.698641] LNet: Removed LNI 192.168.203.5@tcp [ 880.734467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.798143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 880.821495] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 880.825201] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 880.833698] LNet: 50962:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 880.902636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.926442] LNet: 50981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.939992] LNetError: 50981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.948945] LNet: Removed LNI 192.168.203.5@tcp [ 880.954154] LNetError: 50981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 880.969836] LNet: Removed LNI 10.1.2.3@tcp [ 880.990570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.071854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 881.104203] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 881.114598] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 881.120520] LNet: 50989:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 881.330890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.349389] LNet: 51015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.354597] LNetError: 51015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.360151] LNet: Removed LNI 192.168.203.5@tcp [ 881.362629] LNetError: 51015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.372660] LNet: Removed LNI 10.1.2.3@tcp [ 881.386059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.488649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 881.519537] LNet: Added LNI 192.168.203.5@tcp [8/256/0/182] [ 881.526428] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 881.662767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.694311] LNet: 51043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.701282] LNetError: 51043:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.705323] LNet: Removed LNI 192.168.203.5@tcp [ 881.707946] LNetError: 51043:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.721577] LNet: Removed LNI 10.1.2.3@tcp [ 881.741323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.800543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 881.818143] LNet: Added LNI 192.168.203.5@tcp [8/256/0/182] [ 881.822372] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 881.824310] LNet: 51051:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 881.879071] LNet: Removed LNI 192.168.203.5@tcp [ 881.891247] LNet: Removed LNI 10.1.2.3@tcp [ 882.552501] Key type .llcrypt unregistered [ 882.558501] Key type ._llcrypt unregistered [ 892.798644] Key type ._llcrypt registered [ 892.800844] Key type .llcrypt registered [ 893.912567] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:40:49 (1781239249) [ 894.330463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 894.366455] alg: No test for adler32 (adler32-zlib) [ 895.307325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.380376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.418854] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 896.123581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.5 [ 896.145811] LNet: 52042:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 896.158214] LNet: Removed LNI 192.168.203.5@tcp [ 902.376540] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:40:58 (1781239258) [ 902.696893] LNet: 52614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.706432] LNetError: 52614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.722853] LNet: Removed LNI 192.168.203.5@tcp [ 903.347727] Key type .llcrypt unregistered [ 903.351968] Key type ._llcrypt unregistered [ 909.032257] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.264719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.025058] Key type ._llcrypt registered [ 912.027163] Key type .llcrypt registered [ 912.200444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.220300] alg: No test for adler32 (adler32-zlib) [ 913.697589] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.786151] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:41:15 (1781239275) [ 919.914780] LNet: 53518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.920891] LNetError: 53518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.941260] LNet: Removed LNI 10.1.2.3@tcp [ 920.800463] Key type .llcrypt unregistered [ 920.808603] Key type ._llcrypt unregistered [ 926.728902] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 926.953150] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 926.961540] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 926.968745] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 929.684580] Key type ._llcrypt registered [ 929.687309] Key type .llcrypt registered [ 929.929194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 929.948646] alg: No test for adler32 (adler32-zlib) [ 930.867814] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 938.137638] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:41:33 (1781239293) [ 939.785733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 941.345481] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:41:37 (1781239297) [ 941.553628] LNet: 54709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.559192] LNetError: 54709:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 941.585576] LNet: Removed LNI 10.1.2.3@tcp [ 942.171070] Key type .llcrypt unregistered [ 942.174903] Key type ._llcrypt unregistered [ 949.530153] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.820654] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 949.827237] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 949.838647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.435638] Key type ._llcrypt registered [ 951.437695] Key type .llcrypt registered [ 951.708743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 951.726657] alg: No test for adler32 (adler32-zlib) [ 952.610150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.666299] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 960.505206] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:41:56 (1781239316) [ 960.869738] LNet: 55931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.876966] LNetError: 55931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 960.894762] LNet: Removed LNI 10.1.2.3@tcp [ 961.452443] Key type .llcrypt unregistered [ 961.456835] Key type ._llcrypt unregistered [ 967.361777] Key type ._llcrypt registered [ 967.369123] Key type .llcrypt registered [ 967.533864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.553422] alg: No test for adler32 (adler32-zlib) [ 968.466658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 968.489909] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 975.975259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 976.665870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 976.720833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 976.773891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 976.792260] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 977.050219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 977.078847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 977.103186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 977.127895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 977.168989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 977.196229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 977.235492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 977.280910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 977.322550] LNet: 56368:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 977.357488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 977.389426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 977.433564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 977.466897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 977.503445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 977.539290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 977.573993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 978.159174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 978.209583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 978.248966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 978.265898] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 978.476204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 978.513993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 978.555413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 978.598735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 978.646665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 978.716419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 978.720258] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 978.761161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 978.793915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 978.820056] LNet: 56478:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 978.839329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 978.874200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 978.920115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 978.962790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 979.009617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 979.052953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 979.104510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 979.672473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 979.706858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 979.742233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 979.763838] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 979.981561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 980.000569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 980.026041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 980.048133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 980.069213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 980.112460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 980.139457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 980.162992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 980.184162] LNet: 56587:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 980.200561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 980.235997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 980.280386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 980.321346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 980.357369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 980.427550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 980.470871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 981.113571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 981.170415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 981.210711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 981.459235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 981.498895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 981.530177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 981.564132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 981.602780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 981.669313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 981.720425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 981.772922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 981.812113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 981.845199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 981.880781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 981.926367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 981.961914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 982.009402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 982.058705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 982.653553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 982.699212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 982.725985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 982.740477] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 982.752942] LNet: Skipped 1 previous similar message [ 983.052934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 983.086633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 983.120244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 983.157612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 983.203353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 983.235831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 983.269550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 983.318372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 983.371515] LNet: 56805:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 983.385982] LNet: 56805:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 983.443239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 983.493158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 983.536455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 983.581290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 983.617623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 983.672428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 983.720283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 984.343231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 984.395383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 984.430570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 984.714448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 984.757130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 984.803448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 984.844655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 984.886503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 984.931859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 984.962408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 984.995337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 985.017130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 985.052240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 985.090484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 985.126170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 985.185096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 985.230496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 985.279127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 985.571642] LNet: 56955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.583986] LNetError: 56955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.595865] LNet: Removed LNI 192.168.203.5@tcp [ 986.091786] Key type .llcrypt unregistered [ 986.095409] Key type ._llcrypt unregistered [ 1003.250093] Key type ._llcrypt registered [ 1003.256206] Key type .llcrypt registered [ 1004.394307] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:42:40 (1781239360) [ 1005.229385] Key type .llcrypt unregistered [ 1005.237661] Key type ._llcrypt unregistered [ 1012.183352] Key type ._llcrypt registered [ 1012.185730] Key type .llcrypt registered [ 1012.404924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1012.417558] alg: No test for adler32 (adler32-zlib) [ 1013.369315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1013.401569] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1022.900813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1033.908938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1033.973965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1034.018780] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1035.635572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 1035.680811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 1 [ 1035.718962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 1035.750552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 1 [ 1035.788286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 1035.825211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 1 [ 1035.897758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 1035.956323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 1 [ 1036.002503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1036.044224] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1036.052150] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1036.281507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1036.320693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1036.359547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1036.414551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1036.458741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1036.511738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1036.539222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1036.584437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1036.616955] LNet: 58424:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1036.649763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1036.694183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1036.722923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1036.752396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1036.786370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.833813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.886496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1037.339623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 1037.382597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 1 [ 1037.411198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 1037.438995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 1 [ 1037.474823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 1037.518610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 1 [ 1037.562393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 1037.606835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 1 [ 1037.648904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1037.662392] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1037.665347] LNet: Skipped 2 previous similar messages [ 1037.837365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1037.859690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1037.879713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1037.900699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1037.924413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1037.947873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1037.971087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1037.996601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1038.011272] LNet: 58545:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1038.034392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1038.067523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1038.099728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1038.128659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1038.154751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1038.183730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1038.222291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1038.707379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 1038.779405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 1 [ 1038.854862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 1038.889557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 1 [ 1038.923246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 1038.960281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 1 [ 1039.011202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 1039.043479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 1 [ 1039.078249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1039.101247] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1039.107402] LNet: Skipped 2 previous similar messages [ 1039.420154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1039.454553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1039.493621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1039.529922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1039.578285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1039.617246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1039.660183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1039.701918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1039.722847] LNet: 58666:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1039.758995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1039.786685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1039.844497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1039.888400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1039.920409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1039.950480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1039.980236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1040.548786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 1040.593946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 1 [ 1040.634452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 1040.680710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 1 [ 1040.763234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 1040.830602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 1 [ 1040.878651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 1040.913951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 1 [ 1040.941950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1041.140337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1041.170571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1041.227942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1041.276057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1041.316387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1041.338269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1041.377391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1041.407962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1041.459079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1041.503278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1041.548493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1041.587720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1041.637264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1041.686159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1041.727963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1042.351659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 1042.392455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 1 [ 1042.441945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 1042.498334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 1 [ 1042.553198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 1042.580929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 1 [ 1042.616459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 1042.656544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 1 [ 1042.686627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1042.702234] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1042.708276] LNet: Skipped 5 previous similar messages [ 1042.947704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.991435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1043.081583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1043.140860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.190643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.239494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.297664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.357117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.379988] LNet: 58909:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1043.387906] LNet: 58909:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1043.404700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.474508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.541643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.591167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.652356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.690758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.736409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1044.490466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 1044.562484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_error -r 1 [ 1044.624135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 1044.661687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_error -r 1 [ 1044.728581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 1044.783432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_error -r 1 [ 1044.820039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 1044.870665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_error -r 1 [ 1044.905399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1045.225508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1045.253627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1045.297994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1045.319186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1045.359164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1045.391363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1045.480046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1045.533600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1045.590454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1045.642770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1045.692163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1045.734548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1045.772342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1045.846907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1045.926250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1052.315611] LNet: 59106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.320846] LNetError: 59106:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.332705] LNet: Removed LNI 192.168.203.5@tcp [ 1053.353890] LNet: Removed LNI 192.168.203.5@tcp1 [ 1053.811163] Key type .llcrypt unregistered [ 1053.814662] Key type ._llcrypt unregistered [ 1071.942907] Key type ._llcrypt registered [ 1071.944726] Key type .llcrypt registered [ 1073.344664] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:43:49 (1781239429) [ 1074.286425] Key type .llcrypt unregistered [ 1074.293342] Key type ._llcrypt unregistered [ 1082.032282] Key type ._llcrypt registered [ 1082.035948] Key type .llcrypt registered [ 1082.220695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1082.244982] alg: No test for adler32 (adler32-zlib) [ 1083.270211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1083.322219] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1092.735085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1093.628152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 1093.706656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 1093.768717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1093.834938] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1094.176809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1094.234503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1094.320736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.378800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1094.416202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1094.494619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1094.581953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1094.643172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1094.685548] LNet: 60356:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1094.729636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1094.802340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1094.876630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1094.921677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1094.997183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.038485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.096404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1095.934357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 1095.976611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 1096.024983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1096.053018] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1096.290088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1096.314772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1096.338760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1096.361776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1096.394569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1096.431228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1096.457686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1096.483801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1096.498373] LNet: 60465:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1096.513217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1096.540690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1096.564995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1096.603246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1096.654925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1096.687374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1096.725752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1097.388781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 1097.417969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 1097.471898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1097.509138] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1097.837542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.888271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1097.951743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.998446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.030930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.047376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.067071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1098.084911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1098.098639] LNet: 60574:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1098.117566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1098.136528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1098.161298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1098.182497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1098.202500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1098.224409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1098.242930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1098.449579] LNet: 60616:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.464947] LNetError: 60616:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.512240] LNet: Removed LNI 192.168.203.5@tcp [ 1100.003939] Key type .llcrypt unregistered [ 1100.008504] Key type ._llcrypt unregistered [ 1116.220393] Key type ._llcrypt registered [ 1116.223641] Key type .llcrypt registered [ 1117.799503] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:44:33 (1781239473) [ 1118.769373] Key type .llcrypt unregistered [ 1118.786263] Key type ._llcrypt unregistered [ 1127.010154] Key type ._llcrypt registered [ 1127.011738] Key type .llcrypt registered [ 1127.662182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1127.699514] alg: No test for adler32 (adler32-zlib) [ 1128.636891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1128.674612] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1136.302181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1145.617429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1145.680206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1145.717150] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1147.179708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 1147.236587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 1 [ 1147.279330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 1147.357616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 1 [ 1147.429383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 1147.480523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 1 [ 1147.543413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 1147.600310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 1 [ 1147.660204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1147.698851] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1147.706198] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 1147.952923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.972458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1147.987866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1148.023929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.050710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.066520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1148.098936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1148.151706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1148.177855] LNet: 62082:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1148.218789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1148.245508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1148.265825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1148.295898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1148.322776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.359567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1148.399403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1148.448354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.937554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 1148.967368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_error -r 1 [ 1148.989924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 1149.018925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_error -r 1 [ 1149.059778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 1149.094642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_error -r 1 [ 1149.116244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 1149.156065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_error -r 1 [ 1149.190238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1149.206333] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1149.212582] LNet: Skipped 2 previous similar messages [ 1149.364314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.392288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1149.420430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.442864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.457466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.478857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.505379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.530424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.547309] LNet: 62205:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.596976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.638739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.686976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.725369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1149.781256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1149.836421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1149.890886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1149.928718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1150.353577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 1150.389173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_timeout -r 1 [ 1150.430680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 1150.467859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_timeout -r 1 [ 1150.521520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 1150.598142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_timeout -r 1 [ 1150.647833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 1150.688794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_timeout -r 1 [ 1150.723136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 1150.741619] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1150.932684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1150.974947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1151.009354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.032622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.059754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.106869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.142165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.182128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.200179] LNet: 62328:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1151.217178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.242898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.267820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.304350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.339232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.366890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1151.391181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1151.422907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.158377] LNet: 62406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.164720] LNetError: 62406:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.173963] LNet: Removed LNI 192.168.203.5@tcp [ 1156.181819] LNet: Removed LNI 192.168.203.5@tcp1 [ 1156.468709] Key type .llcrypt unregistered [ 1156.470558] Key type ._llcrypt unregistered [ 1174.261712] Key type ._llcrypt registered [ 1174.269735] Key type .llcrypt registered [ 1175.540435] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:45:31 (1781239531) [ 1176.847071] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1178.635253] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:45:34 (1781239534) [ 1179.544432] Key type .llcrypt unregistered [ 1179.547059] Key type ._llcrypt unregistered [ 1186.402500] Key type ._llcrypt registered [ 1186.415979] Key type .llcrypt registered [ 1186.608644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1186.627854] alg: No test for adler32 (adler32-zlib) [ 1187.579360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1187.627211] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1195.220630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1195.915705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 1195.941452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 1195.967509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1195.981485] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1196.207197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.236274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1196.269357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.288795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.317507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.350293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.388267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.418661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.442777] LNet: 63950:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.474308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.513939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.551342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.587820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.612580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.647184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.971507] LNet: 63989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.989681] LNetError: 63989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.026204] LNet: Removed LNI 192.168.203.5@tcp [ 1197.453482] Key type .llcrypt unregistered [ 1197.457705] Key type ._llcrypt unregistered [ 1214.345415] Key type ._llcrypt registered [ 1214.347355] Key type .llcrypt registered [ 1214.566740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1214.592223] alg: No test for adler32 (adler32-zlib) [ 1215.573992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1215.606836] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1222.822228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1231.681339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1231.719278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1231.751289] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1233.139485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 1233.177342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e network_timeout -r 1 [ 1233.229648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 1233.272086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e network_timeout -r 1 [ 1233.317765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 1233.370802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e network_timeout -r 1 [ 1233.409712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 1233.465845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e network_timeout -r 1 [ 1233.502059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1233.528925] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1233.722142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1233.754707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1233.783365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1233.809450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1233.837898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1233.863184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1233.903513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1233.926329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1233.952353] LNet: 64868:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1233.969666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1233.996222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.021773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.052052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.081365] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1234.081401] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1234.090935] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 1234.094516] LNet: Skipped 1 previous similar message [ 1234.115582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.150786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.482650] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1238.796511] LNet: 64941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.803095] LNetError: 64941:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1238.810086] LNet: Removed LNI 192.168.203.5@tcp [ 1238.818940] LNet: Removed LNI 192.168.203.5@tcp1 [ 1239.169398] Key type .llcrypt unregistered [ 1239.176286] Key type ._llcrypt unregistered [ 1254.000220] Key type ._llcrypt registered [ 1254.002680] Key type .llcrypt registered [ 1255.323522] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 00:46:51 (1781239611) [ 1255.659649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1255.668428] alg: No test for adler32 (adler32-zlib) [ 1256.639633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1256.742849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1256.763389] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1257.484393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1257.504751] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1258.282819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1258.359044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1258.416381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1258.457958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1258.503610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1258.537205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1258.569883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 1258.605427] LNetError: 66040:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 1258.720347] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1258.723258] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1260.786209] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1261.030365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1264.864420] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1265.766540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1265.818528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1265.912751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1265.940215] LNet: 66147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.966443] LNetError: 66147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.976163] LNet: Removed LNI 192.168.203.5@tcp [ 1266.981733] LNetError: 66147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.999301] LNet: Removed LNI 192.168.203.5@tcp1 [ 1267.058730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1267.223816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1267.289631] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1268.501986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1268.546132] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1270.007475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1270.072427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.161765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1270.239988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1270.289268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1270.394244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1270.472551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1270.514561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 1270.553986] LNetError: 66420:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 1271.265334] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1274.454110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1277.409170] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1277.421362] LNet: Skipped 1 previous similar message [ 1282.571853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1282.616273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1282.667987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.496722] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 00:47:26 (1781239646) [ 1291.602340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1291.640673] LNet: 67133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.659380] LNetError: 67133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.671474] LNet: Removed LNI 192.168.203.5@tcp [ 1291.683853] LNet: Removed LNI 192.168.203.5@tcp1 [ 1291.729267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1291.784666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1291.814280] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1292.594932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1292.625253] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1293.485074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1293.599639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1293.651223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1293.683788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1293.738389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1293.773381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1293.834944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 0 [ 1293.945925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1294.816675] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.5@tcp [ 1294.817223] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1294.835719] LNet: Skipped 1 previous similar message [ 1308.360983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1308.590995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1308.656526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.719726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 500 [ 1308.811603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1314.393779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1314.492559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1314.534894] LNet: 67614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1314.540531] LNetError: 67614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.547195] LNetError: 67614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1314.552448] LNet: Removed LNI 192.168.203.5@tcp [ 1314.573672] LNet: Removed LNI 192.168.203.5@tcp1 [ 1314.608478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.656300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1314.676523] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1315.515583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1315.536831] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1316.747188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1316.789160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1316.868667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1316.898361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1316.949617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1317.012986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1317.061947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1317.115401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 0 [ 1318.752460] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1318.764422] LNet: Skipped 3 previous similar messages [ 1319.693845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1330.261873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1330.301284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1330.350507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1338.362940] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 00:48:14 (1781239694) [ 1338.932616] LNet: 68672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.938026] LNetError: 68672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1338.944530] LNetError: 68672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1338.950622] LNet: Removed LNI 192.168.203.5@tcp [ 1338.961812] LNet: Removed LNI 192.168.203.5@tcp1 [ 1339.739832] Key type .llcrypt unregistered [ 1339.745250] Key type ._llcrypt unregistered [ 1346.020336] Key type ._llcrypt registered [ 1346.022025] Key type .llcrypt registered [ 1346.323644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1346.352594] alg: No test for adler32 (adler32-zlib) [ 1347.276682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1347.303195] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1357.198021] Lustre: DEBUG MARKER: Initial discovery [ 1357.267499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 1359.945731] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1360.021462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1360.060924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 1360.084635] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1360.117469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1360.146317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.105@tcp -r 1 -m GET -l 3 [ 1360.212478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.105@tcp -r 1 -m GET -e local_error [ 1360.283032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.105@tcp -r 1 -m PUT -l 6 [ 1360.612117] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1361.797137] Lustre: DEBUG MARKER: Force oleg305-client.virtnet to discover 192.168.203.105@tcp (in background) [ 1361.865469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 1363.522276] Lustre: DEBUG MARKER: Force oleg305-server to discover 192.168.203.5@tcp [ 1364.960547] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1366.121647] Lustre: DEBUG MARKER: Wait for 69427 [ 1367.863088] Lustre: DEBUG MARKER: Finished wait on 69427 [ 1367.928299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1367.995315] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1368.011061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1368.386158] LNet: 69884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.394471] LNetError: 69884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.447921] LNet: Removed LNI 192.168.203.5@tcp [ 1370.040269] Key type .llcrypt unregistered [ 1370.042522] Key type ._llcrypt unregistered [ 1384.384787] Key type ._llcrypt registered [ 1384.392532] Key type .llcrypt registered [ 1385.511192] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 00:49:01 (1781239741) [ 1386.214574] Key type .llcrypt unregistered [ 1386.216384] Key type ._llcrypt unregistered [ 1392.114840] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1392.287815] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1392.304861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1392.315434] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1394.891339] Key type ._llcrypt registered [ 1394.895643] Key type .llcrypt registered [ 1395.098889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1395.116535] alg: No test for adler32 (adler32-zlib) [ 1396.078387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.153911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.180243] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1396.934292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1396.964860] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1406.230635] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 00:49:21 (1781239761) [ 1406.758974] LNet: 71964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.771419] LNetError: 71964:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.783239] LNet: Removed LNI 192.168.203.5@tcp [ 1406.799799] LNet: Removed LNI 10.1.2.3@tcp [ 1407.602533] Key type .llcrypt unregistered [ 1407.611178] Key type ._llcrypt unregistered [ 1415.371792] Key type ._llcrypt registered [ 1415.380143] Key type .llcrypt registered [ 1415.690702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1415.729842] alg: No test for adler32 (adler32-zlib) [ 1416.686466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.756164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1416.781802] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1417.708746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1417.745837] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1419.010714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.5@tcp [ 1420.064135] LNetError: Unexpected error -22 connecting to 192.168.203.5@tcp at host 192.168.203.5:988 [ 1426.197926] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 00:49:42 (1781239782) [ 1426.511665] LNet: 73095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.524146] LNetError: 73095:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.528846] LNet: Removed LNI 192.168.203.5@tcp [ 1426.554407] LNet: Removed LNI 10.1.2.3@tcp [ 1427.311413] Key type .llcrypt unregistered [ 1427.315430] Key type ._llcrypt unregistered [ 1434.660116] Key type ._llcrypt registered [ 1434.662770] Key type .llcrypt registered [ 1434.897779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1434.939245] alg: No test for adler32 (adler32-zlib) [ 1435.932344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.991069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1436.023637] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1436.938341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1436.973776] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1437.942495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.5@tcp1 --nid 192.168.203.5@tcp2 [ 1442.852986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp1 192.168.203.5@tcp2 [ 1442.874866] LNetError: 73930:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.5@tcp1 to 192.168.203.5@tcp2 - Route table may be misconfigured [ 1442.904811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp2 192.168.203.5@tcp1 [ 1449.713635] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 00:50:05 (1781239805) [ 1450.223778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.268748] LNet: 74503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.289493] LNetError: 74503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.305667] LNet: Removed LNI 192.168.203.5@tcp1 [ 1451.381065] LNet: Removed LNI 192.168.203.5@tcp2 [ 1451.408080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.499543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1451.535047] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1452.307262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1452.323083] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1453.120154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1453.160422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.5@tcp [ 1453.209367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp1 192.168.203.5@tcp1 [ 1453.247869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.5@tcp -d 192.168.203.5@tcp -e network_timeout [ 1453.283883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -e network_timeout [ 1453.319971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -e network_timeout [ 1453.357625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -e network_timeout [ 1453.412559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1453.434982] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1453.457402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1453.536130] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1453.594288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1453.713769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1460.254745] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 00:50:16 (1781239816) [ 1460.684288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.737951] LNet: 75365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.749941] LNetError: 75365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.776345] LNetError: 75365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1460.784678] LNet: Removed LNI 192.168.203.5@tcp [ 1461.799201] LNet: Removed LNI 192.168.203.5@tcp1 [ 1461.857632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.994080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1462.241553] LNet: 75402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.595857] Key type .llcrypt unregistered [ 1462.598942] Key type ._llcrypt unregistered [ 1474.885570] Key type ._llcrypt registered [ 1474.887735] Key type .llcrypt registered [ 1476.422142] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 00:50:32 (1781239832) [ 1476.919327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1476.944829] alg: No test for adler32 (adler32-zlib) [ 1477.897154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.370115] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1480.924259] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 00:50:37 (1781239837) [ 1481.109301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.151103] LNet: 76499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.181927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.246513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.271783] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1482.072215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1482.107531] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1483.309649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1483.372406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp1 [ 1483.427369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp1 [ 1489.989036] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 00:50:45 (1781239845) [ 1491.908803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg305-server [ 1493.900386] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 00:50:49 (1781239849) [ 1496.094417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg305-server [ 1498.778187] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 00:50:54 (1781239854) [ 1500.616644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg305-server [ 1502.858232] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 00:50:58 (1781239858) [ 1505.304661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg305-server [ 1507.391527] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 00:51:03 (1781239863) [ 1509.418603] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg305-server [ 1511.649452] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 00:51:07 (1781239867) [ 1514.176574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg305-server [ 1516.577413] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 00:51:11 (1781239871) [ 1519.285413] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg305-server [ 1521.432454] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 00:51:16 (1781239876) [ 1523.452550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg305-server [ 1525.173968] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 00:51:21 (1781239881) [ 1527.281182] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg305-server [ 1529.146352] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 00:51:25 (1781239885) [ 1529.756872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1529.792455] LNet: 80379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.806448] LNetError: 80379:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.825049] LNet: Removed LNI 192.168.203.5@tcp [ 1530.857620] LNet: Removed LNI 192.168.203.5@tcp1 [ 1530.906576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.989684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1531.032177] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1531.699418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1531.786333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1531.978290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.032035] LNet: 80526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1532.051434] LNetError: 80526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1532.055721] LNetError: 80526:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1532.078804] LNet: Removed LNI 192.168.203.5@tcp [ 1532.106070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.213939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1532.252822] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1533.048978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1533.339040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.400793] LNetError: 80672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.437565] LNet: Removed LNI 192.168.203.5@tcp [ 1533.479284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.556372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1533.576369] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1534.581644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1534.724591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1534.968426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.000120] LNet: 80819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.007208] LNet: 80819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1536.043823] LNet: Removed LNI 192.168.203.5@tcp [ 1536.079271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.189369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1536.240263] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1537.401621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1537.672399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.699437] LNetError: 80965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1537.707204] LNetError: 80965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1537.731884] LNet: Removed LNI 192.168.203.5@tcp [ 1537.777781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.903574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1537.935575] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1538.854969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1538.976363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1539.334098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.365778] LNet: 81112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.378459] LNet: 81112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1540.466597] LNet: Removed LNI 192.168.203.5@tcp [ 1540.512721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.613479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1540.681877] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1541.645344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1541.956638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.039940] LNetError: 81257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.051977] LNetError: 81257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1543.082880] LNet: Removed LNI 192.168.203.5@tcp [ 1543.130827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.199115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.236464] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1544.008698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1544.138528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1544.473736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.578976] LNet: Removed LNI 192.168.203.5@tcp [ 1545.641894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.779072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1545.819501] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1546.730708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1548.262557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.317339] LNet: 81560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.326606] LNet: 81560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1549.426713] LNet: Removed LNI 192.168.203.5@tcp [ 1549.491188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.553316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.584909] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1550.422466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1550.539504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1552.090584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.112734] LNetError: 81716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.120885] LNetError: 81716:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1552.143210] LNet: Removed LNI 192.168.203.5@tcp [ 1552.167969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.223397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1552.260911] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1553.120758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1553.436360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.534989] LNet: Removed LNI 192.168.203.5@tcp [ 1554.565172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.669616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.702308] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1555.457894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1555.543421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1555.845580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.912283] LNet: Removed LNI 192.168.203.5@tcp [ 1556.949239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.026659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1557.070374] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1558.129443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1558.527951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.594986] LNet: Removed LNI 192.168.203.5@tcp [ 1559.668289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.750113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1559.780732] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1560.379774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1560.428800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.462259] LNet: Removed LNI 192.168.203.5@tcp [ 1560.505026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.588529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1560.644546] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1561.640810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1570.325706] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 00:52:06 (1781239926) [ 1570.560417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.595822] LNet: 82991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.601429] LNet: 82991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1570.609037] LNetError: 82991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.618501] LNetError: 82991:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1570.654720] LNet: Removed LNI 192.168.203.5@tcp [ 1570.680546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.741280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1570.777529] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 1572.174419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.230343] LNet: Removed LNI 192.168.203.5@tcp231 [ 1572.290587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.392092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1572.420518] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 1572.433171] LNet: 83138:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1572.831918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1572.871594] LNet: Removed LNI 192.168.203.5@tcp231 [ 1572.891842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1572.930488] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 1574.347949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.390128] LNet: Removed LNI 192.168.203.5@tcp231 [ 1574.440816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.507430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1574.557925] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/0] [ 1574.568126] LNet: 83291:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1575.127376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1575.161972] LNet: Removed LNI 192.168.203.5@tcp231 [ 1575.195557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1575.247196] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/0] [ 1576.659804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.707475] LNet: Removed LNI 192.168.203.5@tcp231 [ 1576.743233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.832510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1576.875559] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/60] [ 1576.881358] LNet: 83442:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.401369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1577.467839] LNet: Removed LNI 192.168.203.5@tcp231 [ 1577.489430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1577.540646] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/60] [ 1578.864877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.909450] LNet: Removed LNI 192.168.203.5@tcp231 [ 1578.932477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.051783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1579.085950] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 1579.099065] LNet: 83594:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1579.684081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1579.720940] LNet: Removed LNI 192.168.203.5@tcp231 [ 1579.754667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1579.801675] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 1588.876518] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 00:52:24 (1781239944) [ 1589.006629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.101670] LNet: Removed LNI 192.168.203.5@tcp231 [ 1589.157688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.267460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1589.303434] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 1590.176946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1590.244219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1590.295971] LNet: Removed LNI 192.168.203.5@tcp232 [ 1590.324326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1590.375718] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 1591.257525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1591.296348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1591.355901] LNet: Removed LNI 192.168.203.5@tcp232 [ 1591.405126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1591.447640] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 1592.507690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1592.566625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1592.611483] LNet: Removed LNI 192.168.203.5@tcp232 [ 1592.639707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1592.685539] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 1593.492842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1593.526447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1593.549523] LNet: Removed LNI 192.168.203.5@tcp232 [ 1593.562620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1599.840547] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 00:52:35 (1781239955) [ 1601.210600] Key type .llcrypt unregistered [ 1601.213080] Key type ._llcrypt unregistered [ 1607.578506] Key type ._llcrypt registered [ 1607.586432] Key type .llcrypt registered [ 1608.003913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1608.046866] alg: No test for adler32 (adler32-zlib) [ 1609.084224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1609.175224] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1617.821800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1629.190061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.218422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1629.235450] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1630.778728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 2 [ 1630.812326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 2 [ 1630.842086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 2 [ 1630.871794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 2 [ 1630.902471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_interrupt -r 2 [ 1630.938991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 2 [ 1630.971315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_interrupt -r 2 [ 1631.012585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 2 [ 1631.061651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1631.096609] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1631.108676] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1631.410184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1631.463556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1631.510286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1631.571485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1631.613913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1631.647497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1631.681604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1631.732824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1631.758542] LNet: 86097:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1631.804645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1631.843191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1631.889239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1631.940788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1632.008878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1632.052730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1632.601531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 2 [ 1632.642475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 2 [ 1632.687819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 2 [ 1632.762618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 2 [ 1632.838639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_dropped -r 2 [ 1632.913714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 2 [ 1632.964415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_dropped -r 2 [ 1633.001801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 2 [ 1633.035400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1633.052928] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1633.306301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1633.345092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1633.411966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1633.461308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1633.507690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1633.538616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1633.568286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1633.620861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1633.647102] LNet: 86217:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1633.664552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1633.694510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1633.736959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1633.779741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1633.812051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1633.853584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1634.449300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 2 [ 1634.486715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 2 [ 1634.533614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 2 [ 1634.611090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 2 [ 1634.693270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_aborted -r 2 [ 1634.757831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 2 [ 1634.800355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_aborted -r 2 [ 1634.853688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 2 [ 1634.897823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1634.922898] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1635.171138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1635.230517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1635.259556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1635.317030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1635.384434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1635.449516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1635.538765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1635.613470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1635.634403] LNet: 86337:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1635.655445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1635.687236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1635.740588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1635.799484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1635.840524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1635.870847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1636.230670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 2 [ 1636.273639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 2 [ 1636.317253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 2 [ 1636.361065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 2 [ 1636.385293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_no_route -r 2 [ 1636.406948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 2 [ 1636.439409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_no_route -r 2 [ 1636.495832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 2 [ 1636.526890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1636.844135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1636.880246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1636.906711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1636.932462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1636.967349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1637.013424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1637.070390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1637.117962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1637.148504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1637.201088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1637.236405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1637.269388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1637.309375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1637.371942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1637.887811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 2 [ 1637.926120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 2 [ 1637.971886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 2 [ 1638.044695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 2 [ 1638.096303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_timeout -r 2 [ 1638.150225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 2 [ 1638.182218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_timeout -r 2 [ 1638.231460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 2 [ 1638.272438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1638.313192] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 1638.328416] LNet: Skipped 1 previous similar message [ 1638.616984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1638.647548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1638.681525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1638.711811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1638.760465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1638.799538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1638.844931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1638.910938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1638.968195] LNet: 86578:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1638.978766] LNet: 86578:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1639.018452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1639.056204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1639.086777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1639.116202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1639.146272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1639.177211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1639.609319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 2 [ 1639.647686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 2 [ 1639.696449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 2 [ 1639.739917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 2 [ 1639.777970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_dropped -r 2 [ 1639.812578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 2 [ 1639.868549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_dropped -r 2 [ 1639.924564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 2 [ 1639.978797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 1640.026540] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 1640.319749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1640.347749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1640.370419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1640.395292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1640.425439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1640.446614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1640.482269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1640.512555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1640.538619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1640.575627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1640.609735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1640.637228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1640.666631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1640.694780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1645.667560] LNet: 86772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.694216] LNetError: 86772:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.709119] LNet: Removed LNI 192.168.203.5@tcp [ 1645.726457] LNet: Removed LNI 192.168.203.5@tcp1 [ 1646.052190] Key type .llcrypt unregistered [ 1646.055285] Key type ._llcrypt unregistered [ 1662.107367] Key type ._llcrypt registered [ 1662.123812] Key type .llcrypt registered [ 1663.870207] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 00:53:39 (1781240019) [ 1664.426159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1664.442376] alg: No test for adler32 (adler32-zlib) [ 1665.422268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.505226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.538935] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1666.279287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1666.301277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 1666.313606] LNetError: 87752:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 1666.463543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.5@tcp [ 1666.604865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1672.919846] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 00:53:48 (1781240028) [ 1673.253601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1673.295256] LNet: 88341:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1673.307789] LNetError: 88341:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1673.328400] LNet: Removed LNI 192.168.203.5@tcp [ 1673.364401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1673.476387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1673.545910] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1674.297428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1674.317528] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1675.450187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.5@tcp [ 1675.508030] LNetError: 88607:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 1676.512434] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 1676.513922] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 1676.791949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.5@tcp [ 1686.128969] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 00:54:01 (1781240041) [ 1688.345237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg305-server [ 1690.352156] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 00:54:06 (1781240046) [ 1692.125505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg305-server [ 1693.658369] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 00:54:09 (1781240049) [ 1695.557071] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg305-server [ 1697.341682] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 00:54:13 (1781240053) [ 1697.537843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.585890] LNet: 90238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.606729] LNetError: 90238:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.629836] LNet: Removed LNI 192.168.203.5@tcp [ 1697.660887] LNet: Removed LNI 192.168.203.5@tcp1 [ 1697.680524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.734959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.766947] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1698.494617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1698.548966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1698.594906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1698.695126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1698.754779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1698.817313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1705.504710] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 00:54:21 (1781240061) [ 1706.204304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.221881] LNet: 90965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.230603] LNetError: 90965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.239987] LNetError: 90965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1706.256486] LNet: Removed LNI 192.168.203.5@tcp [ 1706.281983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.322908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1706.342478] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1713.913942] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 00:54:29 (1781240069) [ 1714.305775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.341176] LNet: 91668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.345980] LNetError: 91668:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.359405] LNet: Removed LNI 192.168.203.5@tcp [ 1714.383227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.472896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1714.505062] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1722.163415] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 00:54:37 (1781240077) [ 1723.984476] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1726.125153] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 00:54:41 (1781240081) [ 1726.831805] LNet: 92704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.842096] LNetError: 92704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.860368] LNet: Removed LNI 192.168.203.5@tcp [ 1727.399689] Key type .llcrypt unregistered [ 1727.401283] Key type ._llcrypt unregistered [ 1733.453688] Key type ._llcrypt registered [ 1733.457348] Key type .llcrypt registered [ 1733.758985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1733.802550] alg: No test for adler32 (adler32-zlib) [ 1734.713225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1734.759258] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1742.040265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1746.780293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.105@tcp [ 1747.131955] LNet: 93122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1747.148646] LNetError: 93122:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1747.179214] LNet: Removed LNI 192.168.203.5@tcp [ 1747.833685] Key type .llcrypt unregistered [ 1747.837119] Key type ._llcrypt unregistered [ 1759.578194] Key type ._llcrypt registered [ 1759.581329] Key type .llcrypt registered [ 1761.019392] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1763.295635] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1764.996597] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 00:55:20 (1781240120) [ 1766.764290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1768.571890] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 00:55:24 (1781240124) [ 1771.548435] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg305-server [ 1773.560141] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 00:55:29 (1781240129) [ 1775.383352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg305-server [ 1777.207278] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 00:55:33 (1781240133) [ 1778.444256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1778.468896] alg: No test for adler32 (adler32-zlib) [ 1779.532461] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 1779.793881] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1781.520509] Key type lgssc registered [ 1783.108397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1805.617561] Key type lgssc unregistered [ 1806.029495] LNet: 96301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.047977] LNetError: 96301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.076638] LNet: Removed LNI 192.168.203.5@tcp [ 1806.772555] Key type .llcrypt unregistered [ 1806.779358] Key type ._llcrypt unregistered [ 1823.122714] Key type ._llcrypt registered [ 1823.129958] Key type .llcrypt registered [ 1824.388248] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 00:56:20 (1781240180) [ 1824.884526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.911795] alg: No test for adler32 (adler32-zlib) [ 1825.891636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.942414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1825.998878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1826.054329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1826.099916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.757931] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 00:56:28 (1781240188) [ 1832.862552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1832.880876] LNet: 97711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.902084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.981947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.001659] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1833.889849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1833.913044] LNet: 97860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.919735] LNetError: 97860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.935315] LNet: Removed LNI 192.168.203.5@tcp [ 1833.951948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.046299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1834.068618] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1834.642724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.668730] LNetError: 97888:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.686150] LNet: Removed LNI 192.168.203.5@tcp [ 1834.710772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.755383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1834.800121] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1841.921396] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 00:56:37 (1781240197) [ 1842.022449] LNet: 98470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.038945] LNet: 98470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1842.058186] LNetError: 98470:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.095727] LNet: Removed LNI 192.168.203.5@tcp [ 1842.862919] Key type .llcrypt unregistered [ 1842.867799] Key type ._llcrypt unregistered [ 1849.984354] Key type ._llcrypt registered [ 1849.986338] Key type .llcrypt registered [ 1850.238592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1850.269959] alg: No test for adler32 (adler32-zlib) [ 1851.240474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.333811] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1851.336252] LNet: *** cfs_fail_loc=e003, val=100*** [ 1852.042881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1852.058766] LNet: *** cfs_fail_loc=e003, val=100*** [ 1852.061417] LNet: Skipped 2 previous similar messages [ 1852.067960] Ping reply received with mapping: [ 1852.070219] found metadata reply NI[192.168.203.5@tcp] [ 1852.072872] 05 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1852.077613] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1852.081302] 5a 5a 5a 5a 5a 5a 5a 5a [ 1852.196791] LNet: 98876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.204661] LNetError: 98876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.227827] LNet: Removed LNI 192.168.203.5@tcp [ 1853.907474] Key type .llcrypt unregistered [ 1853.910955] Key type ._llcrypt unregistered [ 1864.554729] Key type ._llcrypt registered [ 1864.557508] Key type .llcrypt registered [ 1865.920332] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 00:57:01 (1781240221) [ 1866.567799] Key type .llcrypt unregistered [ 1866.580130] Key type ._llcrypt unregistered [ 1873.025158] Key type ._llcrypt registered [ 1873.026761] Key type .llcrypt registered [ 1873.307616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1873.332304] alg: No test for adler32 (adler32-zlib) [ 1874.580157] LNetError: 99922:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1875.185456] Key type .llcrypt unregistered [ 1875.188852] Key type ._llcrypt unregistered [ 1880.343145] Key type ._llcrypt registered [ 1880.346950] Key type .llcrypt registered [ 1881.681874] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 00:57:17 (1781240237) [ 1882.483429] Key type .llcrypt unregistered [ 1882.490405] Key type ._llcrypt unregistered [ 1888.346822] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1888.355598] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1891.361899] Key type ._llcrypt registered [ 1891.363545] Key type .llcrypt registered [ 1891.666628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1891.699548] alg: No test for adler32 (adler32-zlib) [ 1892.762187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1892.911760] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1904.210462] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 00:57:39 (1781240259) [ 1904.347742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.382660] LNet: 101698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.389223] LNetError: 101698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1904.410680] LNet: Removed LNI 10.1.2.3@tcp [ 1904.482710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.321708] Key type .llcrypt unregistered [ 1905.325545] Key type ._llcrypt unregistered [ 1911.473563] Key type ._llcrypt registered [ 1911.477817] Key type .llcrypt registered [ 1911.691719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1911.725725] alg: No test for adler32 (adler32-zlib) [ 1912.768970] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1912.849314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1914.238937] LNet: 102000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.247476] LNetError: 102000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.272515] LNet: Removed LNI 192.168.203.5@tcp [ 1915.044348] Key type .llcrypt unregistered [ 1915.049635] Key type ._llcrypt unregistered [ 1927.474658] Key type ._llcrypt registered [ 1927.482041] Key type .llcrypt registered [ 1929.012037] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 00:58:04 (1781240284) [ 1929.632753] Key type .llcrypt unregistered [ 1929.637735] Key type ._llcrypt unregistered [ 1936.036963] Key type ._llcrypt registered [ 1936.040387] Key type .llcrypt registered [ 1936.230576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1936.243388] alg: No test for adler32 (adler32-zlib) [ 1944.084256] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 00:58:19 (1781240299) [ 1944.192506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.231725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.281540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1944.317956] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1944.968827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1944.999064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1945.031218] LNet: Removed LNI 192.168.203.5@tcp [ 1945.243961] LNet: 103744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.659603] Key type .llcrypt unregistered [ 1945.661479] Key type ._llcrypt unregistered [ 1958.716299] Key type ._llcrypt registered [ 1958.718179] Key type .llcrypt registered [ 1960.024802] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 00:58:35 (1781240315) [ 1960.502695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1960.521307] alg: No test for adler32 (adler32-zlib) [ 1961.481292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1961.553725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1961.572686] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1962.353762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1962.387801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.5@tcp [ 1962.412265] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1962.427046] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1962.431558] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1962, 0, 0 [ 1962.442059] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1962.448184] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1962.451701] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1962.455231] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1962, 0, 0 [ 1962.460253] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1962.463143] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1962.473194] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1962.480584] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1962, 0, 2012 [ 1962.491654] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1962.500654] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1962.508451] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1962.512967] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1962, 0, 2012 [ 1962.522069] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1969.604317] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 00:58:45 (1781240325) [ 1970.245518] LNet: 105285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.258055] LNetError: 105285:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1971.302271] LNet: Removed LNI 192.168.203.5@tcp [ 1972.245682] Key type .llcrypt unregistered [ 1972.252803] Key type ._llcrypt unregistered [ 1979.109513] Key type ._llcrypt registered [ 1979.111494] Key type .llcrypt registered [ 1979.407487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1979.455485] alg: No test for adler32 (adler32-zlib) [ 1980.400416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1980.430781] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1989.364495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1998.541080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1998.586375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1998.610520] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1999.890787] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2000.160814] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2000.168699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2000.177772] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2002.571463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2002.601494] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2013.706339] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 00:59:29 (1781240369) [ 2013.889413] LNet: 106796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.894908] LNetError: 106796:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.944206] LNet: Removed LNI 192.168.203.5@tcp [ 2014.946400] LNetError: 106796:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.961591] LNet: Removed LNI 192.168.203.5@tcp1 [ 2014.983596] LNet: Removed LNI 10.1.2.3@tcp99 [ 2015.750614] Key type .llcrypt unregistered [ 2015.757659] Key type ._llcrypt unregistered [ 2023.017109] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2023.024337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2025.547144] Key type ._llcrypt registered [ 2025.549609] Key type .llcrypt registered [ 2025.768843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2025.792506] alg: No test for adler32 (adler32-zlib) [ 2026.765522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2026.856996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2026.907465] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2027.688826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2027.734348] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2030.056862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2030.140528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2030.180805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2030.195557] LNetError: 107541:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2030.226607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2030.255647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2030.828412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2030.925161] LNet: 107561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.933964] LNetError: 107561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.968343] LNet: Removed LNI 192.168.203.5@tcp [ 2031.975476] LNetError: 107561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2031.996458] LNet: Removed LNI 10.1.2.3@tcp [ 2032.652691] Key type .llcrypt unregistered [ 2032.660065] Key type ._llcrypt unregistered [ 2044.457935] Key type ._llcrypt registered [ 2044.460137] Key type .llcrypt registered [ 2045.736802] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:00:01 (1781240401) [ 2046.190962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2046.211312] alg: No test for adler32 (adler32-zlib) [ 2047.149632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.220070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2047.250817] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2048.032902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 2048.120366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg305-client@tcp [ 2054.667526] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:00:10 (1781240410) [ 2054.779828] LNet: 109086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.808818] LNetError: 109086:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.845814] LNet: Removed LNI 192.168.203.5@tcp [ 2055.598358] Key type .llcrypt unregistered [ 2055.599786] Key type ._llcrypt unregistered [ 2061.767793] Key type ._llcrypt registered [ 2061.774965] Key type .llcrypt registered [ 2062.027180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.051793] alg: No test for adler32 (adler32-zlib) [ 2063.046562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2063.076753] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2063.101575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2070.168966] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:00:26 (1781240426) [ 2071.551443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2073.031869] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:00:29 (1781240429) [ 2074.650675] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2076.710468] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:00:32 (1781240432) [ 2076.873801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2076.889426] LNet: 110515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.895376] LNetError: 110515:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.931176] LNet: Removed LNI 192.168.203.5@tcp [ 2076.949030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.036775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2077.066077] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2078.151520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2078.233156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2085.786328] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:00:41 (1781240441) [ 2086.092316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2086.120726] LNet: 111225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.131046] LNetError: 111225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.158079] LNet: Removed LNI 192.168.203.5@tcp [ 2086.207812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.296297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2086.333381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2086.978444] Key type .llcrypt unregistered [ 2086.981415] Key type ._llcrypt unregistered [ 2097.493697] Key type ._llcrypt registered [ 2097.496561] Key type .llcrypt registered [ 2099.035925] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:00:54 (1781240454) [ 2099.458626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2099.481873] alg: No test for adler32 (adler32-zlib) [ 2100.459308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.571473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2100.609264] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2101.315422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2101.392841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2108.676872] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:01:04 (1781240464) [ 2108.776968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2108.811974] LNet: 112789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.823602] LNetError: 112789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.841774] LNet: Removed LNI 192.168.203.5@tcp [ 2108.885264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.993848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2109.026801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2115.793362] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:01:11 (1781240471) [ 2115.940586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.011845] LNet: 113371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.038473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.164428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2116.937292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2117.000302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2117.066921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2117.601808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2117.675137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2117.809813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.832379] LNet: 113409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.867919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.971935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2118.405333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2118.440943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2118.531842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2119.104597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2119.159582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2125.946154] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:01:21 (1781240481) [ 2126.137753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.164183] LNet: 114008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.210469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.258718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2126.285331] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2126.906505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2126.924186] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2127.679974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2127.741678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2127.971062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.999772] LNetError: 114289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2128.012294] LNet: Removed LNI 192.168.203.5@tcp [ 2128.030640] LNet: Removed LNI 192.168.203.5@tcp2 [ 2128.067482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.145100] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2128.156947] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2128.164168] LNet: 114295:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2128.799751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2129.003377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.024263] LNetError: 114332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2129.031200] LNetError: 114332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2129.038757] LNet: Removed LNI 192.168.203.5@tcp [ 2129.057961] LNet: Removed LNI 192.168.203.5@tcp2 [ 2129.095363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.181439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2129.236910] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2129.971800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2129.988330] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2131.236700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2131.326908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2131.424873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2131.763353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.786370] LNet: 114617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.798968] LNet: 114617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2131.809114] LNetError: 114617:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.813508] LNetError: 114617:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2131.827044] LNet: Removed LNI 192.168.203.5@tcp [ 2132.846422] LNet: Removed LNI 192.168.203.5@tcp2 [ 2132.893815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.999508] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2133.005833] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2133.013605] LNet: 114623:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2133.770212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2134.033224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.058538] LNet: Removed LNI 192.168.203.5@tcp [ 2135.080836] LNet: Removed LNI 192.168.203.5@tcp2 [ 2135.122912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.193514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2135.257086] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2136.140479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2136.188291] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2137.117526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2137.191665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2137.430957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.476078] LNetError: 114940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.486299] LNetError: 114940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2137.498222] LNet: Removed LNI 192.168.203.5@tcp [ 2137.526744] LNet: Removed LNI 192.168.203.5@tcp2 [ 2137.548675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.631529] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2137.639030] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2137.642959] LNet: 114946:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2138.274628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2138.503754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.616621] LNet: Removed LNI 192.168.203.5@tcp [ 2139.641634] LNet: Removed LNI 192.168.203.5@tcp2 [ 2139.676793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.754906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.772709] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2140.454628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2140.473602] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2141.293483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2141.341801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2141.408708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2141.611712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.632651] LNet: 115267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.637831] LNet: 115267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2142.688389] LNet: Removed LNI 192.168.203.5@tcp [ 2143.718654] LNet: Removed LNI 192.168.203.5@tcp2 [ 2143.768664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.878324] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2143.892831] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2143.907619] LNet: 115274:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2144.571528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2144.860939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.897775] LNet: Removed LNI 192.168.203.5@tcp [ 2145.965806] LNet: Removed LNI 192.168.203.5@tcp2 [ 2145.995686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.042355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2146.095505] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2147.131125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2147.159642] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2148.190883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2148.252466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2148.330892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2148.739554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.779039] LNetError: 115608:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.786854] LNetError: 115608:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2148.805965] LNet: Removed LNI 192.168.203.5@tcp [ 2148.832050] LNet: Removed LNI 192.168.203.5@tcp2 [ 2148.866264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.006918] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2149.016559] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2149.023579] LNet: 115614:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2149.596872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2150.059744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.136212] LNet: Removed LNI 192.168.203.5@tcp [ 2151.148487] LNet: Removed LNI 192.168.203.5@tcp2 [ 2151.175500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.217729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2151.235422] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2151.960216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2151.994872] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2153.097230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2153.161394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2153.213382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2153.297279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2153.744785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.784494] LNet: Removed LNI 192.168.203.5@tcp [ 2155.826592] LNet: Removed LNI 192.168.203.5@tcp2 [ 2155.904204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.953989] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2155.962436] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 2156.412760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2166.608557] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2168.157362] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:02:04 (1781240524) [ 2168.521613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.541991] LNet: 116734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.547540] LNet: 116734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2168.553622] LNetError: 116734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.559027] LNetError: 116734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2168.564340] LNet: Removed LNI 192.168.203.5@tcp [ 2169.579438] LNet: Removed LNI 192.168.203.5@tcp2 [ 2169.644205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.182854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2170.228467] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2171.506810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2171.561095] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2172.698886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2172.726710] LNet: Removed LNI 192.168.203.5@tcp [ 2172.756403] LNet: Removed LNI 10.1.2.3@tcp [ 2172.827107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.936156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2172.976333] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2172.980240] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2173.731119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2173.780085] LNet: Removed LNI 192.168.203.5@tcp [ 2173.805057] LNet: Removed LNI 10.1.2.3@tcp [ 2180.787957] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:02:16 (1781240536) [ 2181.064444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.110711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.194195] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2181.301644] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2181.313300] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2181.316512] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2183.904647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2183.945361] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2184.673802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2184.686989] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2186.792166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 2186.850612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2186.943646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.5@tcp -m PUT -e local_timeout -r 1 [ 2187.036327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2187.120656] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2187.126277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2187.468920] LNetError: 117674:0:(lib-move.c:4710:lnet_parse()) 192.168.203.5@tcp, src 192.168.203.5@tcp: Dropping PUT (error -108 looking up sender) [ 2189.536367] LNet: Removed LNI 192.168.203.5@tcp [ 2189.552354] LNet: Removed LNI 10.1.2.3@tcp [ 2189.943677] Key type .llcrypt unregistered [ 2189.947530] Key type ._llcrypt unregistered [ 2203.571414] Key type ._llcrypt registered [ 2203.573253] Key type .llcrypt registered [ 2205.127768] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:02:40 (1781240560) [ 2205.645483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2205.670110] alg: No test for adler32 (adler32-zlib) [ 2206.642978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.103380] LNet: 118987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.744772] Key type .llcrypt unregistered [ 2207.748824] Key type ._llcrypt unregistered [ 2214.312819] Key type ._llcrypt registered [ 2214.316471] Key type .llcrypt registered [ 2214.486780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2214.503649] alg: No test for adler32 (adler32-zlib) [ 2215.525778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2215.564805] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2223.495485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 2223.730250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2224.625363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.105@tcp [ 2224.800666] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 2224.977433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2225.009700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2225.036231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2225.068382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2225.096560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2225.157823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2225.236987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2225.306412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2225.358116] LNet: 119438:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2225.406332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2225.465492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2225.557698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2225.657859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2225.740748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2225.830804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2226.902568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2227.239380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2227.297903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2227.348827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2227.399339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2227.497230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2227.562580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2227.607200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2227.648994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2227.666802] LNet: 119555:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2227.694737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2227.730256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2227.761515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2227.795940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2227.826585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2227.898789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2228.874076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.105@tcp [ 2229.195709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2229.235781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2229.277689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2229.322898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2229.370216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2229.407937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2229.452868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2229.506877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2229.540070] LNet: 119673:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2229.565873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2229.608557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2229.643446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2229.694931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2229.746663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2229.807694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2230.672921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2231.084401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2231.136049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2231.170956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2231.210446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2231.240338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2231.296643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2231.352747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2231.404903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2231.449996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2231.503987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2231.536954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2231.612057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2231.650234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2231.679716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2232.051218] LNet: 119836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.060332] LNetError: 119836:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2232.092627] LNet: Removed LNI 192.168.203.5@tcp [ 2232.629114] Key type .llcrypt unregistered [ 2232.630739] Key type ._llcrypt unregistered [ 2249.292497] Key type ._llcrypt registered [ 2249.295707] Key type .llcrypt registered [ 2250.623955] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:03:26 (1781240606) [ 2251.023305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2251.039551] alg: No test for adler32 (adler32-zlib) [ 2252.012018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.502640] LNet: 120724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.280377] Key type .llcrypt unregistered [ 2253.284209] Key type ._llcrypt unregistered [ 2260.572308] Key type ._llcrypt registered [ 2260.574667] Key type .llcrypt registered [ 2260.738745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2260.771445] alg: No test for adler32 (adler32-zlib) [ 2261.739905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2261.772261] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2270.026268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 2279.344517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.415541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2279.457098] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 2280.963916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2281.798905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.105@tcp [ 2282.197436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2282.228496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2282.262511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2282.272442] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 2282.320710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2282.371257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2282.414744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2282.464338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2282.518416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2282.568306] LNet: 121377:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2282.618182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2282.668253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2282.713207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2282.764201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2282.812446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2282.861752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2283.695565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2284.087907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2284.152600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2284.213862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2284.308359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2284.415500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2284.493799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2284.571825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2284.620752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2284.650372] LNet: 121494:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2284.690988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2284.763853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2284.847888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2284.920048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2284.991622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2285.061649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2286.147800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.105@tcp [ 2286.432348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2286.461590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2286.496905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2286.529309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2286.563656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2286.604310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2286.642732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2286.687288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2286.704410] LNet: 121612:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2286.725963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2286.769203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2286.823054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2286.883909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2286.947492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2287.004391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2287.713608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2287.962779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2288.001714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2288.068117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2288.135533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2288.213778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2288.282636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2288.359484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2288.380958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2288.405534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2288.452682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2288.547512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2288.615924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2288.686716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2288.732768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2293.893841] LNet: 121810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.921168] LNetError: 121810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.935144] LNet: Removed LNI 192.168.203.5@tcp [ 2293.953956] LNet: Removed LNI 192.168.203.5@tcp1 [ 2294.441686] Key type .llcrypt unregistered [ 2294.446579] Key type ._llcrypt unregistered [ 2317.671525] Key type ._llcrypt registered [ 2317.673820] Key type .llcrypt registered [ 2319.134934] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:04:34 (1781240674) [ 2328.026762] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:04:43 (1781240683) [ 2337.702963] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:04:53 (1781240693) [ 2345.497681] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:05:01 (1781240701) [ 2353.776175] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:05:09 (1781240709) [ 2360.620280] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:05:16 (1781240716) [ 2368.078679] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:05:24 (1781240724) [ 2375.918550] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:05:31 (1781240731) [ 2384.278429] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:05:39 (1781240739) [ 2391.500958] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:05:47 (1781240747) [ 2400.010667] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:05:55 (1781240755) [ 2408.444962] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:06:04 (1781240764) [ 2415.913996] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:06:11 (1781240771) [ 2423.614085] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:06:19 (1781240779) [ 2431.501360] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:06:27 (1781240787) [ 2440.080968] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:06:35 (1781240795) [ 2448.423187] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:06:44 (1781240804) [ 2457.091758] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:06:52 (1781240812) [ 2457.547664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2457.572463] alg: No test for adler32 (adler32-zlib) [ 2458.515069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2458.708635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2458.754125] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2460.029504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2460.058757] LNet: 132530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2460.070531] LNetError: 132530:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2460.088417] LNet: Removed LNI 192.168.203.5@tcp [ 2460.136128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2460.218672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2460.247036] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2467.359834] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:07:03 (1781240823) [ 2475.498888] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:07:11 (1781240831) [ 2483.314243] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:07:19 (1781240839) [ 2491.915600] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:07:27 (1781240847) [ 2499.138306] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:07:35 (1781240855) [ 2507.206901] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:07:42 (1781240862) [ 2515.234036] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:07:51 (1781240871) [ 2523.158389] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:07:58 (1781240878) [ 2531.050117] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:08:06 (1781240886) [ 2538.317475] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:08:14 (1781240894) [ 2545.892612] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:08:21 (1781240901) [ 2553.769542] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:08:29 (1781240909) [ 2561.977195] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2331 sec ========== 01:08:37 (1781240917) [ 2562.369512] LNet: 139994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2562.374402] LNetError: 139994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2562.393585] LNet: Removed LNI 192.168.203.5@tcp [ 2563.075801] Key type .llcrypt unregistered [ 2563.081376] Key type ._llcrypt unregistered [ 2577.083914] Key type ._llcrypt registered [ 2577.087699] Key type .llcrypt registered [ 2577.714602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2577.742992] alg: No test for adler32 (adler32-zlib) [ 2578.929278] Lustre: Lustre: Build Version: 2.17.53_78_gc17e35b [ 2579.194597] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 2580.889549] Key type lgssc registered [ 2582.501990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2650.081152] Lustre: Mounted lustre-client [ 2655.267062] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2664.059197] Lustre: Unmounted lustre-client [ 2705.142300] Key type lgssc unregistered [ 2705.529188] LNet: 142928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2705.539089] LNetError: 142928:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2705.577589] LNet: Removed LNI 192.168.203.5@tcp [ 2706.541207] Key type .llcrypt unregistered [ 2706.543548] Key type ._llcrypt unregistered