[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 432096345 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003084] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008686] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011127] LSM: Security Framework initializing [ 0.012056] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.015031] *** VALIDATE selinux *** [ 0.023641] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027624] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032050] *** VALIDATE tmpfs *** [ 0.033456] *** VALIDATE proc *** [ 0.034260] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.036276] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040031] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.043762] debug: unmapping init [mem 0xffffffff9b259000-0xffffffff9b260fff] [ 0.045904] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046716] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047027] ... version: 2 [ 0.048015] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053014] ... event mask: 000000070000000f [ 0.054302] rcu: Hierarchical SRCU implementation. [ 0.056437] smp: Bringing up secondary CPUs ... [ 0.057560] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.062169] smp: Brought up 1 node, 4 CPUs [ 0.064018] smpboot: Max logical packages: 1 [ 0.065013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136943] node 0 deferred pages initialised in 67ms [ 0.139154] devtmpfs: initialized [ 0.140253] x86/mm: Memory block size: 128MB [ 0.142907] gcov: version magic: 0x41383552 [ 0.144088] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146249] pinctrl core: initialized pinctrl subsystem [ 0.147221] [ 0.147868] ************************************************************* [ 0.148017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149013] ** ** [ 0.150015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151013] ** ** [ 0.152011] ** This means that this kernel is built to expose internal ** [ 0.153064] ** IOMMU data structures, which may compromise security on ** [ 0.154014] ** your system. ** [ 0.155016] ** ** [ 0.156012] ** If you see this message and you are not debugging the ** [ 0.157014] ** kernel, report this immediately to your vendor! ** [ 0.158013] ** ** [ 0.159015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160013] ************************************************************* [ 0.161675] NET: Registered protocol family 16 [ 0.162515] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165629] cpuidle: using governor menu [ 0.167953] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.171464] PCI: Using configuration type 1 for base access [ 0.173121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.185309] cryptd: max_cpu_qlen set to 1000 [ 0.187091] ACPI: Added _OSI(Module Device) [ 0.188041] ACPI: Added _OSI(Processor Device) [ 0.189018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.197226] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201549] ACPI: Interpreter enabled [ 0.203060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.204014] ACPI: Using IOAPIC for interrupt routing [ 0.207110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.211513] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223304] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.226042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.228020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.232070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.236169] acpiphp: Slot [2] registered [ 0.237101] acpiphp: Slot [5] registered [ 0.238092] acpiphp: Slot [6] registered [ 0.240127] acpiphp: Slot [3] registered [ 0.241117] acpiphp: Slot [4] registered [ 0.243115] acpiphp: Slot [7] registered [ 0.244103] acpiphp: Slot [8] registered [ 0.246117] acpiphp: Slot [9] registered [ 0.248128] acpiphp: Slot [10] registered [ 0.249130] acpiphp: Slot [11] registered [ 0.250186] acpiphp: Slot [12] registered [ 0.252149] acpiphp: Slot [13] registered [ 0.254108] acpiphp: Slot [14] registered [ 0.256110] acpiphp: Slot [15] registered [ 0.257183] acpiphp: Slot [16] registered [ 0.259117] acpiphp: Slot [17] registered [ 0.261103] acpiphp: Slot [18] registered [ 0.262130] acpiphp: Slot [19] registered [ 0.264110] acpiphp: Slot [20] registered [ 0.266124] acpiphp: Slot [21] registered [ 0.267109] acpiphp: Slot [22] registered [ 0.269112] acpiphp: Slot [23] registered [ 0.271107] acpiphp: Slot [24] registered [ 0.272132] acpiphp: Slot [25] registered [ 0.274107] acpiphp: Slot [26] registered [ 0.276116] acpiphp: Slot [27] registered [ 0.277108] acpiphp: Slot [28] registered [ 0.279179] acpiphp: Slot [29] registered [ 0.281108] acpiphp: Slot [30] registered [ 0.283119] acpiphp: Slot [31] registered [ 0.284155] PCI host bridge to bus 0000:00 [ 0.286031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.289027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.291022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.294024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.297025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.300032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.302177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.305946] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.309347] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.317504] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.322056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.325019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.328022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.331019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.333649] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.336783] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.340044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.342895] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.347924] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.357668] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.362014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.368769] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.377014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.381984] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.398016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.408629] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.415013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.421014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.435015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.449858] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.452471] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.455379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.458375] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.460178] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.465110] iommu: Default domain type: Passthrough [ 0.467441] SCSI subsystem initialized [ 0.469180] ACPI: bus type USB registered [ 0.471085] usbcore: registered new interface driver usbfs [ 0.473080] usbcore: registered new interface driver hub [ 0.475087] usbcore: registered new device driver usb [ 0.477180] pps_core: LinuxPPS API ver. 1 registered [ 0.480013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.483057] PTP clock support registered [ 0.486047] EDAC MC: Ver: 3.0.0 [ 0.488243] PCI: Using ACPI for IRQ routing [ 0.489759] NetLabel: Initializing [ 0.491012] NetLabel: domain hash size = 128 [ 0.493010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.495076] NetLabel: unlabeled traffic allowed by default [ 0.497120] vgaarb: loaded [ 0.499191] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.501011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.507004] clocksource: Switched to clocksource kvm-clock [ 0.611332] VFS: Disk quotas dquot_6.6.0 [ 0.613148] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.616333] *** VALIDATE ramfs *** [ 0.617791] *** VALIDATE hugetlbfs *** [ 0.619745] pnp: PnP ACPI init [ 0.622699] pnp: PnP ACPI: found 6 devices [ 0.658317] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.661802] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.663547] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.665808] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.668446] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.671054] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.674092] NET: Registered protocol family 2 [ 0.676496] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.681599] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.685433] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.691072] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.694473] TCP: Hash tables configured (established 65536 bind 65536) [ 0.697731] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.701188] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.704652] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.708188] NET: Registered protocol family 1 [ 0.710881] RPC: Registered named UNIX socket transport module. [ 0.713451] RPC: Registered udp transport module. [ 0.715582] RPC: Registered tcp transport module. [ 0.717454] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.720357] NET: Registered protocol family 44 [ 0.722278] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.724742] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.727570] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.730442] PCI: CLS 0 bytes, default 64 [ 0.733153] Unpacking initramfs... [ 2.131894] debug: unmapping init [mem 0xffff8f96fcc64000-0xffff8f96fffcffff] [ 2.135951] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.138114] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.141587] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.657441] Initialise system trusted keyrings [ 2.659274] Key type blacklist registered [ 2.661789] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.671203] zbud: loaded [ 2.675273] *** VALIDATE nfs *** [ 2.676768] *** VALIDATE nfs4 *** [ 2.678782] pstore: using deflate compression [ 2.682556] Platform Keyring initialized [ 2.797611] NET: Registered protocol family 38 [ 2.800096] Key type asymmetric registered [ 2.801837] Asymmetric key parser 'x509' registered [ 2.803790] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.808474] io scheduler mq-deadline registered [ 2.810035] io scheduler kyber registered [ 2.811617] io scheduler bfq registered [ 2.813493] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.816412] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.819089] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.821816] ACPI: Power Button [PWRF] [ 2.827378] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.833881] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.848481] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.877261] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.907060] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.914158] Non-volatile memory driver v1.3 [ 2.915619] Linux agpgart interface v0.103 [ 2.950232] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.954023] vda: detected capacity change from 0 to 76791808 [ 2.978100] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.981748] vdb: detected capacity change from 0 to 1073741824 [ 2.988766] libphy: Fixed MDIO Bus: probed [ 2.993860] usbcore: registered new interface driver usbserial_generic [ 2.996878] usbserial: USB Serial support registered for generic [ 2.999347] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.003637] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.005752] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.008455] mousedev: PS/2 mouse device common for all mice [ 3.012533] rtc_cmos 00:05: RTC can wake from S4 [ 3.015745] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.020224] rtc_cmos 00:05: registered as rtc0 [ 3.022501] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.023390] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.029366] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.030888] intel_pstate: CPU model not supported [ 3.038189] hid: raw HID events driver (C) Jiri Kosina [ 3.040594] usbcore: registered new interface driver usbhid [ 3.043120] usbhid: USB HID core driver [ 3.044909] drop_monitor: Initializing network drop monitor service [ 3.047670] Initializing XFRM netlink socket [ 3.050146] NET: Registered protocol family 10 [ 3.053645] Segment Routing with IPv6 [ 3.055738] NET: Registered protocol family 17 [ 3.058756] mpls_gso: MPLS GSO support [ 3.065804] RAS: Correctable Errors collector initialized. [ 3.068178] AVX version of gcm_enc/dec engaged. [ 3.069962] AES CTR mode by8 optimization enabled [ 3.148122] sched_clock: Marking stable (3148100479, 0)->(4128705882, -980605403) [ 3.152554] registered taskstats version 1 [ 3.154837] Loading compiled-in X.509 certificates [ 3.157685] zswap: loaded using pool lzo/zbud [ 3.184870] Key type big_key registered [ 3.200285] Key type encrypted registered [ 3.201984] ima: No TPM chip found, activating TPM-bypass! [ 3.204269] ima: Allocated hash algorithm: sha1 [ 3.206537] ima: No architecture policies found [ 3.208505] evm: Initialising EVM extended attributes: [ 3.210600] evm: security.selinux [ 3.212007] evm: security.ima [ 3.213266] evm: security.capability [ 3.214734] evm: HMAC attrs: 0x1 [ 3.217292] rtc_cmos 00:05: setting system clock to 2026-09-16 16:16:56 UTC (1789575416) [ 3.223928] debug: unmapping init [mem 0xffffffff9c203000-0xffffffff9c3fffff] [ 3.227273] debug: unmapping init [mem 0xffffffff9af82000-0xffffffff9b258fff] [ 3.238275] Write protecting the kernel read-only data: 28672k [ 3.242227] debug: unmapping init [mem 0xffffffff99603000-0xffffffff997fffff] [ 3.246128] debug: unmapping init [mem 0xffffffff99f14000-0xffffffff99ffffff] [ 3.276706] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.285402] systemd[1]: Detected virtualization kvm. [ 3.287156] systemd[1]: Detected architecture x86-64. [ 3.289131] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.315397] systemd[1]: No hostname configured. [ 3.317339] systemd[1]: Set hostname to . [ 3.319589] random: systemd: uninitialized urandom read (16 bytes read) [ 3.322446] systemd[1]: Initializing machine ID from random generator. [ 3.373764] random: ln: uninitialized urandom read (6 bytes read) [ 3.475038] random: systemd: uninitialized urandom read (16 bytes read) [ 3.477873] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.483460] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.489496] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.068949] device-mapper: uevent: version 1.0.3 [ 4.071516] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.714166] virtio_net virtio0 ens2: renamed from eth0 [ 4.773571] scsi host0: ata_piix [ 4.805918] scsi host1: ata_piix [ 4.813927] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.817067] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.376322] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.720482] random: crng init done [ 9.721900] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.983455] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.058512] printk: systemd: 19 output lines suppressed due to ratelimiting [ 11.335932] SELinux: Disabled at runtime. [ 11.404856] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.414180] systemd[1]: Detected virtualization kvm. [ 11.416256] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.916277] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.919819] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.926168] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.932279] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.934738] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.940499] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.944836] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... Starting udev Coldplug all Devices... [ 12.092961] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.319482] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.699414] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.722550] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.802380] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.819722] EDAC sbridge: Ver: 1.1.2 [ 15.597996] Key type dns_resolver registered [ 16.366083] NFS: Registering the id_resolver key type [ 16.385370] Key type id_resolver registered [ 16.389435] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-client login: [ 77.623562] libcfs: loading out-of-tree module taints kernel. [ 77.990624] Key type ._llcrypt registered [ 78.064157] Key type .llcrypt registered [ 78.671013] hrtimer: interrupt took 5966612 ns [ 78.691304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 78.706787] alg: No test for adler32 (adler32-zlib) [ 80.294843] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 81.345913] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 83.207971] Key type lgssc registered [ 85.194173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 218.841525] Lustre: Mounted lustre-client - version 2.17.58_39_gf14a6ef [ 223.748576] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.134642] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing check_logdir /tmp/testlogs/ [ 244.698324] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing yml_node [ 244.703206] Lustre: lustre-OST0000-osc-ffff8f976100f800: disconnect after 23s idle [ 249.394762] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 252.201195] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 254.973317] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 257.217774] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 16 12:21:08 EDT 2026 [ 274.834201] Lustre: Unmounted lustre-client [ 297.652051] Key type lgssc unregistered [ 297.946477] LNet: 8406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 297.957977] LNetError: 8406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 297.976589] LNet: Removed LNI 192.168.203.45@tcp [ 298.719410] Key type .llcrypt unregistered [ 298.724287] Key type ._llcrypt unregistered [ 324.248538] Key type ._llcrypt registered [ 324.251233] Key type .llcrypt registered [ 324.443719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 324.467547] alg: No test for adler32 (adler32-zlib) [ 325.428102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 325.496377] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 325.540216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 329.592367] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 329.739936] LNet: 9285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.745197] LNetError: 9285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 329.761445] LNet: Removed LNI 192.168.203.45@tcp [ 330.339385] Key type .llcrypt unregistered [ 330.344160] Key type ._llcrypt unregistered [ 336.683219] Key type ._llcrypt registered [ 336.687027] Key type .llcrypt registered [ 337.839913] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:22:29 (1789575749) [ 338.349992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 338.368985] alg: No test for adler32 (adler32-zlib) [ 339.303574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.359776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.392174] LNet: 9694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.414280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.495788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.516047] LNet: 9701:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 346.546372] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:22:38 (1789575758) [ 346.637538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.662413] LNet: 10273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.687299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.801203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 353.772378] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:22:45 (1789575765) [ 353.881076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.899442] LNet: 10859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.918086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.016340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 360.426484] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:22:52 (1789575772) [ 360.686844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.715218] LNet: 11453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.766345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.861026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 367.929588] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:22:59 (1789575779) [ 368.094642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.142460] LNet: 12035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.209868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.313964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 369.213448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 376.973942] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:23:08 (1789575788) [ 377.375804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.410509] LNet: 12628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.430375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.580705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 378.237963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 378.856616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 387.190495] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:23:18 (1789575798) [ 387.705457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.751868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.858150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 395.130772] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:23:26 (1789575806) [ 395.266057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.297403] LNet: 13804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.305208] LNet: 13804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 395.366571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.441641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 395.475956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.032908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 396.077963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 396.665896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 396.704071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 397.190598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 397.236604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 397.752938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 397.800253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 398.167582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 398.213982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.805950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 398.848822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 399.431658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 399.503344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 400.114686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 400.166918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 400.766854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 400.812970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 401.366071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 401.419413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 401.974673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 402.037683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 402.509614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 402.580250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 410.501506] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:23:42 (1789575822) [ 410.675177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.773949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.882961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 410.932107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 418.466743] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:23:50 (1789575830) [ 418.604652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.675635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.787245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 418.833870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 426.200650] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:23:58 (1789575838) [ 426.462564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.522623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.588693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 426.632477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 433.531509] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:24:05 (1789575845) [ 433.786893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.819352] LNet: 16251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 433.825259] LNet: 16251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 433.858194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.975442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 434.037335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 441.512091] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:24:13 (1789575853) [ 441.607310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.664737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.758562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 441.809819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 449.170223] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:24:20 (1789575860) [ 449.449806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.517854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.641296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 449.724356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 456.602797] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:24:28 (1789575868) [ 456.911493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.955896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.044129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 457.507356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.117554] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:24:36 (1789575876) [ 464.427743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.473485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.744400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.070489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.841760] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:24:43 (1789575883) [ 472.125599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.168647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.298851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 472.494638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 479.642712] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:24:51 (1789575891) [ 479.888951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.949356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.530126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 480.828436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 488.776326] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:25:00 (1789575900) [ 488.917813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.976262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.100158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 489.177932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.193406] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:25:09 (1789575909) [ 497.327636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.418660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.561037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.623189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.536200] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:25:17 (1789575917) [ 505.803590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.829466] LNet: 21694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.835531] LNet: 21694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 505.856861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.944704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 513.597992] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:25:24 (1789575924) [ 513.696850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.758439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.917193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 522.354637] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:25:34 (1789575934) [ 522.573279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.613989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.750283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 523.482459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 532.873037] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:25:44 (1789575944) [ 532.956060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.981448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.064641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 533.120500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 540.950226] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:25:52 (1789575952) [ 541.227860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.290829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.378774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 541.417606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 549.797907] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:26:01 (1789575961) [ 550.136859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.239322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.346384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 550.382331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 559.298771] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:26:11 (1789575971) [ 559.598690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.638241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.720861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 559.796892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 567.399170] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:26:18 (1789575978) [ 567.501466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.549446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.637072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 567.788385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 567.820949] LNetError: 25812:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 567.882585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 567.979372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 575.274496] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:26:26 (1789575986) [ 575.376832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.434350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.619379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.18dqcaEETf [ 575.722267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.18dqcaEETf [ 575.812813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.18dqcaEETf [ 583.118893] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:26:34 (1789575994) [ 583.242044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.331149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.412116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 583.462364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 583.527235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 590.708465] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:26:42 (1789576002) [ 590.800197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.883133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.029463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 591.063051] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 593.184653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 594.593653] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 596.440071] Key type lgssc registered [ 597.985748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.792355] LNetError: 27574:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.45@tcp99 has no available nets [ 604.528197] Key type lgssc unregistered [ 604.987808] LNetError: 28681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.059900] LNet: Removed LNI 192.168.203.45@tcp [ 606.778399] Key type .llcrypt unregistered [ 606.800045] Key type ._llcrypt unregistered [ 613.714429] Key type ._llcrypt registered [ 613.716258] Key type .llcrypt registered [ 615.261395] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:27:06 (1789576026) [ 616.114297] Key type .llcrypt unregistered [ 616.116549] Key type ._llcrypt unregistered [ 623.236937] Key type ._llcrypt registered [ 623.243290] Key type .llcrypt registered [ 623.441860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 623.471983] alg: No test for adler32 (adler32-zlib) [ 624.463515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.533737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 624.552919] LNet: 29592:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 624.559704] LNet: 29592:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 624.571322] LNet: 29592:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 624.599649] LNet: Message forwarding will be enabled in 2s [ 627.679454] LNet: Message forwarding enabled [ 628.249975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 628.303640] LNet: Message forwarding will stop in 2s to 6s [ 630.761474] LNet: Message forwarding disabled [ 631.834083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 631.852801] LNet: 29682:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 631.858309] LNet: 29682:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 631.864638] LNet: 29682:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 631.875678] LNet: Message forwarding will be enabled in 2s [ 632.003417] LNet: 29695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.539534] Key type .llcrypt unregistered [ 632.542195] Key type ._llcrypt unregistered [ 645.111569] Key type ._llcrypt registered [ 645.114906] Key type .llcrypt registered [ 646.315007] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:27:38 (1789576058) [ 647.037550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 647.050671] alg: No test for adler32 (adler32-zlib) [ 648.055410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.163493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 654.746535] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:27:46 (1789576066) [ 655.024175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.088661] LNet: 31095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.131661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.199577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 655.234844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 655.278994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 655.330977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 655.392521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 655.459154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 655.515985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 655.571255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 655.642550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 655.698951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 655.765852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 655.816140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 655.887134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 655.956928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 656.008112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 656.067818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 656.134726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 656.200937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 656.231313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 656.255368] LNetError: 31138:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 663.118313] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:27:54 (1789576074) [ 663.225654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.266604] LNet: 31704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.290104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.384911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.974898] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:28:02 (1789576082) [ 673.596054] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg345-server [ 675.594059] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:28:07 (1789576087) [ 675.784530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.824404] LNet: 32641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.846250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.920808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 675.960739] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 676.912296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[46-50]@tcp [ 676.943959] LNet: 32774:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 677.028651] LNetError: 32645:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.49@tcp is being used as a gateway but routing feature is not turned on [ 677.054195] LNetError: 32645:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.49@tcp has gone from up to down [ 684.208721] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:28:15 (1789576095) [ 686.331281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg345-server [ 687.951637] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:28:19 (1789576099) [ 688.183498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.213809] LNet: 33712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.221147] LNetError: 33712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.246512] LNet: Removed LNI 192.168.203.45@tcp [ 688.272232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.348201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.378724] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 688.984924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[46-51/2]@tcp [ 689.006487] LNet: 33829:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 689.012442] LNet: 33829:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 689.024511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[46-51/2]@tcp [ 695.869030] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:28:27 (1789576107) [ 696.269794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.296855] LNet: 34423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.305814] LNetError: 34423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.328191] LNet: Removed LNI 192.168.203.45@tcp [ 696.389696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.508205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 696.593829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.663551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.752025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 696.791133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.819408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.876841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 696.948269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.987694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.055255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.143867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.187289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.248841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 697.334877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.380578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.437179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.536654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.568627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.623661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 697.702830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.744270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.805837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.870797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.897765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.939242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 697.999720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.033687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.085314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 698.185574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.245289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.302843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 698.359880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.449674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.529476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 705.600873] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:28:37 (1789576117) [ 707.706866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg345-server [ 709.795567] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:28:41 (1789576121) [ 712.052980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg345-server [ 714.253645] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:28:45 (1789576125) [ 714.536538] LNet: 35825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.545111] LNet: 35825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 715.361628] Key type .llcrypt unregistered [ 715.365175] Key type ._llcrypt unregistered [ 722.975512] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 723.186543] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 723.206393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 723.215209] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 725.817350] Key type ._llcrypt registered [ 725.819731] Key type .llcrypt registered [ 726.059048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.088162] alg: No test for adler32 (adler32-zlib) [ 727.063733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.140191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.189499] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 728.012545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 728.028006] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.985369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 730.009806] LNet: Removed LNI 10.1.2.3@tcp [ 730.118254] LNet: 36537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.139344] LNetError: 36537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.164740] LNet: Removed LNI 192.168.203.45@tcp [ 730.900234] Key type .llcrypt unregistered [ 730.904620] Key type ._llcrypt unregistered [ 742.665135] Key type ._llcrypt registered [ 742.671202] Key type .llcrypt registered [ 744.047987] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:29:15 (1789576155) [ 744.599473] Key type .llcrypt unregistered [ 744.604702] Key type ._llcrypt unregistered [ 751.151249] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 751.455256] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 751.472477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 751.485423] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 753.040752] Key type ._llcrypt registered [ 753.043568] Key type .llcrypt registered [ 753.255765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.284782] alg: No test for adler32 (adler32-zlib) [ 754.273485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.348983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.401585] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 755.136359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 755.160522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 757.718551] LNet: 38019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.724474] LNetError: 38019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 757.735891] LNet: Removed LNI 10.1.2.3@tcp [ 757.757209] LNet: Removed LNI 192.168.203.45@tcp [ 758.358492] Key type .llcrypt unregistered [ 758.362447] Key type ._llcrypt unregistered [ 771.440735] Key type ._llcrypt registered [ 771.442938] Key type .llcrypt registered [ 772.730412] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:29:44 (1789576184) [ 773.261265] Key type .llcrypt unregistered [ 773.264945] Key type ._llcrypt unregistered [ 780.743021] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 781.008474] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 781.027593] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 781.038725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 783.736230] Key type ._llcrypt registered [ 783.739924] Key type .llcrypt registered [ 784.051690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.093773] alg: No test for adler32 (adler32-zlib) [ 785.050835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.101747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 785.147135] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 787.312540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 787.332908] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 789.187438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 789.204391] LNet: Removed LNI 10.1.2.3@tcp [ 789.226130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 789.266676] LNet: Removed LNI 10.1.2.31@tcp [ 789.288378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 789.340379] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.295163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 790.327077] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 791.236898] LNet: 39936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.243874] LNetError: 39936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 791.248684] LNet: Removed LNI 10.1.2.31@tcp [ 791.261836] LNet: Removed LNI 10.1.2.3@tcp [ 791.817611] Key type .llcrypt unregistered [ 791.819051] Key type ._llcrypt unregistered [ 804.885805] Key type ._llcrypt registered [ 804.889166] Key type .llcrypt registered [ 806.407993] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:30:17 (1789576217) [ 806.808085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.826551] alg: No test for adler32 (adler32-zlib) [ 807.839684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.920664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 807.951672] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 808.702228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.726048] LNet: 40886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.730400] LNetError: 40886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.741296] LNet: Removed LNI 192.168.203.45@tcp [ 808.772958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.840531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.45@tcp [ 808.878080] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 809.576388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.45@tcp [ 809.620203] LNet: Removed LNI 192.168.203.45@tcp [ 809.712992] LNet: 41021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.546664] Key type .llcrypt unregistered [ 810.549968] Key type ._llcrypt unregistered [ 821.972362] Key type ._llcrypt registered [ 821.976449] Key type .llcrypt registered [ 823.497196] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:30:35 (1789576235) [ 825.296688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg345-server [ 827.180746] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:30:39 (1789576239) [ 827.693865] Key type .llcrypt unregistered [ 827.701542] Key type ._llcrypt unregistered [ 835.053040] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 835.219609] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 835.231046] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 835.253279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 837.905804] Key type ._llcrypt registered [ 837.908384] Key type .llcrypt registered [ 838.137338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.174744] alg: No test for adler32 (adler32-zlib) [ 839.133757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.219398] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 840.422953] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 843.496594] LNet: 42853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.506993] LNetError: 42853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 843.511800] LNet: Removed LNI 192.168.203.45@tcp1 [ 843.527980] LNet: Removed LNI 10.1.2.3@tcp2 [ 844.703766] Key type .llcrypt unregistered [ 844.705369] Key type ._llcrypt unregistered [ 859.440179] Key type ._llcrypt registered [ 859.441746] Key type .llcrypt registered [ 860.659598] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:31:12 (1789576272) [ 861.151889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 861.189214] alg: No test for adler32 (adler32-zlib) [ 862.273595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.387664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 862.427092] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 863.306534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.362276] LNet: 43821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.376325] LNetError: 43821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.410836] LNet: Removed LNI 192.168.203.45@tcp [ 863.451516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.525666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 863.551020] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 870.314586] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:31:22 (1789576282) [ 870.541798] LNet: 44412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.549514] LNetError: 44412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.563484] LNet: Removed LNI 192.168.203.45@tcp [ 871.125376] Key type .llcrypt unregistered [ 871.132392] Key type ._llcrypt unregistered [ 877.347542] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 877.582668] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 877.585948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 877.589918] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 880.143451] Key type ._llcrypt registered [ 880.146329] Key type .llcrypt registered [ 880.373028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 880.393391] alg: No test for adler32 (adler32-zlib) [ 881.360958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.438163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.470802] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 882.448650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 882.469258] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 884.694346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.718863] LNet: 45151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.727151] LNetError: 45151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 884.743193] LNet: Removed LNI 10.1.2.3@tcp [ 884.784082] LNet: Removed LNI 192.168.203.45@tcp [ 884.833361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.920265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 884.963976] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 884.971866] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 885.479935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.497647] LNet: 45178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.503899] LNetError: 45178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 885.508959] LNetError: 45178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 885.516705] LNet: Removed LNI 10.1.2.3@tcp [ 885.525820] LNet: Removed LNI 192.168.203.45@tcp [ 885.550250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.627719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 885.669607] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 885.682605] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 886.326866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 886.357316] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 887.537466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.568570] LNet: 45340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.583038] LNetError: 45340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 887.594844] LNetError: 45340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 887.604511] LNet: Removed LNI 10.1.2.3@tcp [ 887.611505] LNet: Removed LNI 192.168.203.45@tcp [ 887.626141] LNet: Removed LNI 192.168.203.45@tcp2 [ 887.642181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.723426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 887.765612] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 887.775801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 887.783607] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 896.661779] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:31:47 (1789576307) [ 896.804604] LNet: 45931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.812608] LNetError: 45931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 896.819992] LNetError: 45931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 896.829889] LNet: Removed LNI 10.1.2.3@tcp [ 896.833727] LNet: Removed LNI 192.168.203.45@tcp [ 896.870831] LNet: Removed LNI 192.168.203.45@tcp2 [ 898.058253] Key type .llcrypt unregistered [ 898.060901] Key type ._llcrypt unregistered [ 905.159579] Key type ._llcrypt registered [ 905.163880] Key type .llcrypt registered [ 905.463936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 905.489390] alg: No test for adler32 (adler32-zlib) [ 906.487156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.577894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 906.654902] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 907.809910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.836930] LNet: 46331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.847773] LNetError: 46331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.874188] LNet: Removed LNI 192.168.203.45@tcp [ 907.903996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.962728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 907.994716] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 908.510923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.540433] LNet: 46362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.555907] LNetError: 46362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.605793] LNet: Removed LNI 192.168.203.45@tcp [ 908.642928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.736221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 908.770334] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 909.257933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.308758] LNet: Removed LNI 192.168.203.45@tcp [ 909.329504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.427156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 909.450313] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 910.264106] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.600039] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.616657] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.628484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 913.177381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.196671] LNet: 46472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.203192] LNet: 46472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 913.210021] LNetError: 46472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.216868] LNetError: 46472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 913.230699] LNet: Removed LNI 192.168.203.45@tcp [ 913.274396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.392198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 913.434062] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 921.500438] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:32:13 (1789576333) [ 921.711097] LNet: 47065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.721856] LNetError: 47065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.763646] LNet: Removed LNI 192.168.203.45@tcp [ 922.493408] Key type .llcrypt unregistered [ 922.505382] Key type ._llcrypt unregistered [ 929.046306] Key type ._llcrypt registered [ 929.052272] Key type .llcrypt registered [ 929.238254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 929.266765] alg: No test for adler32 (adler32-zlib) [ 930.214123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.324523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 930.357679] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 931.317586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.338726] LNet: 47465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.344380] LNetError: 47465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.355630] LNet: Removed LNI 192.168.203.45@tcp [ 931.376825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.434272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 931.455921] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 938.544506] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:32:30 (1789576350) [ 938.639769] LNet: 48049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.648225] LNetError: 48049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.681631] LNet: Removed LNI 192.168.203.45@tcp [ 939.556750] Key type .llcrypt unregistered [ 939.564738] Key type ._llcrypt unregistered [ 945.982069] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 946.332525] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 946.341815] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 946.350470] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 949.045780] Key type ._llcrypt registered [ 949.048185] Key type .llcrypt registered [ 949.363165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 949.391821] alg: No test for adler32 (adler32-zlib) [ 950.419391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.501917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.550987] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 951.791034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.810884] LNet: 48500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.823103] LNetError: 48500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.838751] LNet: Removed LNI 192.168.203.45@tcp [ 951.861697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.967847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 952.028291] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 958.533322] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:32:50 (1789576370) [ 958.812656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.840509] LNet: 49098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.851425] LNetError: 49098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.875484] LNet: Removed LNI 192.168.203.45@tcp [ 958.900585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.027155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 964.434804] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:32:56 (1789576376) [ 964.694192] LNet: 49681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.383842] Key type .llcrypt unregistered [ 965.387175] Key type ._llcrypt unregistered [ 970.994259] Key type ._llcrypt registered [ 970.996775] Key type .llcrypt registered [ 971.193696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 971.206240] alg: No test for adler32 (adler32-zlib) [ 972.104730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.160775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 972.187751] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 972.883808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 972.898685] LNet: 50075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.903545] LNetError: 50075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.921178] LNet: Removed LNI 192.168.203.45@tcp [ 972.949697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.039851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 973.084040] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 980.848371] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:33:12 (1789576392) [ 981.159682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.216310] LNet: 50669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.231917] LNetError: 50669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.250914] LNet: Removed LNI 192.168.203.45@tcp [ 981.282654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.369892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 987.431822] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:33:19 (1789576399) [ 989.127325] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 991.176875] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:33:22 (1789576402) [ 992.493457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 994.908457] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:33:26 (1789576406) [ 996.542694] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 998.539219] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:33:30 (1789576410) [ 1000.353699] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1002.817260] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:33:34 (1789576414) [ 1004.738367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1006.575893] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:33:38 (1789576418) [ 1006.853698] LNet: 52694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.623877] Key type .llcrypt unregistered [ 1007.628436] Key type ._llcrypt unregistered [ 1014.715435] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1014.970708] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1014.980743] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1014.989610] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1017.562961] Key type ._llcrypt registered [ 1017.565284] Key type .llcrypt registered [ 1017.864671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1017.879578] alg: No test for adler32 (adler32-zlib) [ 1018.885137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.977826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1019.023633] LNet: Added LNI 192.168.203.45@tcp [8/512/0/180] [ 1019.034842] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1019.040944] LNet: 53011:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1019.153540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.168438] LNet: 53030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.174786] LNetError: 53030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1019.180093] LNet: Removed LNI 10.1.2.3@tcp [ 1019.181858] LNetError: 53030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.196317] LNet: Removed LNI 192.168.203.45@tcp [ 1019.223681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.323343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1019.353785] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1019.438848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.460511] LNet: 53055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.466824] LNetError: 53055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.496219] LNet: Removed LNI 192.168.203.45@tcp [ 1019.523317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.621989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1019.655798] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1019.661052] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1019.665802] LNet: 53063:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1019.761864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1019.797896] LNet: 53081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.809252] LNetError: 53081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1019.816791] LNet: Removed LNI 10.1.2.3@tcp [ 1019.819281] LNetError: 53081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.848550] LNet: Removed LNI 192.168.203.45@tcp [ 1019.887596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.008085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.069389] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1020.076911] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1020.079124] LNet: 53089:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1020.290790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.321272] LNet: 53115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.328844] LNetError: 53115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.341078] LNet: Removed LNI 10.1.2.3@tcp [ 1020.343357] LNetError: 53115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.360733] LNet: Removed LNI 192.168.203.45@tcp [ 1020.381201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.443570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.468615] LNet: Added LNI 192.168.203.45@tcp [8/256/0/182] [ 1020.473706] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1020.566196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.582231] LNet: 53143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.588347] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.594254] LNet: Removed LNI 10.1.2.3@tcp [ 1020.596833] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.612080] LNet: Removed LNI 192.168.203.45@tcp [ 1020.667342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.757612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1020.784606] LNet: Added LNI 192.168.203.45@tcp [8/256/0/182] [ 1020.791918] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1020.795910] LNet: 53151:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1020.898523] LNet: Removed LNI 10.1.2.3@tcp [ 1020.912070] LNet: Removed LNI 192.168.203.45@tcp [ 1021.594664] Key type .llcrypt unregistered [ 1021.596249] Key type ._llcrypt unregistered [ 1033.057932] Key type ._llcrypt registered [ 1033.060510] Key type .llcrypt registered [ 1034.401893] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:34:06 (1789576446) [ 1036.228859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1038.375174] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:34:09 (1789576449) [ 1038.740283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1038.767223] alg: No test for adler32 (adler32-zlib) [ 1039.670678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.714507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1039.730737] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1040.503100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.45 [ 1040.535245] LNet: 54412:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1040.553131] LNet: Removed LNI 192.168.203.45@tcp [ 1047.124587] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:34:19 (1789576459) [ 1047.328442] LNet: 54983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.335153] LNetError: 54983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.365326] LNet: Removed LNI 192.168.203.45@tcp [ 1048.033472] Key type .llcrypt unregistered [ 1048.035373] Key type ._llcrypt unregistered [ 1054.968408] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1055.255979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1058.054749] Key type ._llcrypt registered [ 1058.062498] Key type .llcrypt registered [ 1058.281413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1058.294915] alg: No test for adler32 (adler32-zlib) [ 1059.667674] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1067.183307] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:34:38 (1789576478) [ 1067.344594] LNet: 55886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.358830] LNetError: 55886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1067.402070] LNet: Removed LNI 10.1.2.3@tcp [ 1067.961489] Key type .llcrypt unregistered [ 1067.968842] Key type ._llcrypt unregistered [ 1075.197988] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1075.549988] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1075.554422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1075.562735] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1078.506348] Key type ._llcrypt registered [ 1078.508233] Key type .llcrypt registered [ 1078.737749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1078.773674] alg: No test for adler32 (adler32-zlib) [ 1079.815762] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1086.975127] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:34:58 (1789576498) [ 1088.984528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1091.374869] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:35:02 (1789576502) [ 1091.492620] LNet: 57068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.502361] LNetError: 57068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1091.523579] LNet: Removed LNI 10.1.2.3@tcp [ 1092.491309] Key type .llcrypt unregistered [ 1092.497627] Key type ._llcrypt unregistered [ 1098.539085] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1098.840737] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1098.846751] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1098.858793] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1100.469584] Key type ._llcrypt registered [ 1100.472247] Key type .llcrypt registered [ 1100.797662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1100.828826] alg: No test for adler32 (adler32-zlib) [ 1101.779975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.829886] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1110.509903] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:35:22 (1789576522) [ 1110.988989] LNet: 58296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.993599] LNetError: 58296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1111.007305] LNet: Removed LNI 10.1.2.3@tcp [ 1111.832769] Key type .llcrypt unregistered [ 1111.836552] Key type ._llcrypt unregistered [ 1118.993715] Key type ._llcrypt registered [ 1118.997477] Key type .llcrypt registered [ 1119.258249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1119.294784] alg: No test for adler32 (adler32-zlib) [ 1120.233506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1120.268123] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1128.529488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1129.196565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_interrupt -r 1 [ 1129.251814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_interrupt -r 1 [ 1129.303982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1129.342477] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1129.503228] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1129.604681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.633553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1129.671786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1129.708488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.758567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1129.803334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.852192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.902596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.944038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.966845] LNet: 58733:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.996961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.081414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.189758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.221926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.265740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.302886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.386633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1131.128278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_dropped -r 1 [ 1131.151858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_dropped -r 1 [ 1131.179589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1131.195113] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1131.441106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1131.479905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1131.520584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1131.557688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1131.596598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1131.629937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.667089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.700857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.722154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.739981] LNet: 58844:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.759544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.783676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.851292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1131.900697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1131.942580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.027712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.062606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1132.539974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_aborted -r 1 [ 1132.584680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_aborted -r 1 [ 1132.612402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1132.627773] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1132.853639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.903520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.936438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.964769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.007887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1133.050798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.083975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.112982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.137080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.155234] LNet: 58955:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.170051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.201554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.252569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.327770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.365207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.396210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.421427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1133.991038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_no_route -r 1 [ 1134.022891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_no_route -r 1 [ 1134.067788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1134.331164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.363603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.394362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.435651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.474508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1134.526748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1134.565423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1134.598735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1134.639111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1134.664547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1134.707538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1134.740066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1134.783518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1134.818510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1134.851438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.891488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1135.407081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_timeout -r 1 [ 1135.456532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_timeout -r 1 [ 1135.501717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1135.530438] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1135.536643] LNet: Skipped 1 previous similar message [ 1135.853084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1135.890930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1135.924433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1135.978465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1136.030917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1136.080828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1136.138199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1136.169941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1136.204529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1136.232530] LNet: 59178:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1136.240409] LNet: 59178:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1136.257678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.297895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.347854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.377228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.413093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1136.466237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1136.509545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1137.203718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_error -r 1 [ 1137.259225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_error -r 1 [ 1137.303197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1137.602735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.645686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.703648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.761224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.830303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1137.869361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.904274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.930053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.957791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.997113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.029442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.077957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.115308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.157248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.186926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.216363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1138.467516] LNet: 59330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.475591] LNetError: 59330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.502162] LNet: Removed LNI 192.168.203.45@tcp [ 1139.060781] Key type .llcrypt unregistered [ 1139.065106] Key type ._llcrypt unregistered [ 1157.956109] Key type ._llcrypt registered [ 1157.959108] Key type .llcrypt registered [ 1159.303834] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:36:11 (1789576571) [ 1160.674589] Key type .llcrypt unregistered [ 1160.676123] Key type ._llcrypt unregistered [ 1167.653510] Key type ._llcrypt registered [ 1167.656522] Key type .llcrypt registered [ 1167.928288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1167.952681] alg: No test for adler32 (adler32-zlib) [ 1168.893036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1168.930516] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1176.569989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1187.103767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1187.191135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1187.229119] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1188.742521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_interrupt -r 1 [ 1188.782761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 1 [ 1188.819600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_interrupt -r 1 [ 1188.857370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 1 [ 1188.901618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_interrupt -r 1 [ 1188.945717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 1 [ 1189.023331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_interrupt -r 1 [ 1189.114627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 1 [ 1189.159821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1189.186819] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1189.194915] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1189.442669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.489408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.531199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.579443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.633647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1189.677919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.724720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.757988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.802768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.848562] LNet: 60796:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1189.871483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.912221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.959989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.013332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.074074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.130447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.187649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1190.799692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_dropped -r 1 [ 1190.832777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 1 [ 1190.875217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_dropped -r 1 [ 1190.945446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 1 [ 1191.009683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_dropped -r 1 [ 1191.053102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 1 [ 1191.090688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_dropped -r 1 [ 1191.123872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 1 [ 1191.173775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1191.220896] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1191.231742] LNet: Skipped 2 previous similar messages [ 1191.543765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.595802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1191.670255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.736748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.784396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1191.820972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.862492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.915893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.962810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.991935] LNet: 60920:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.044828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.111770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.165820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.224930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.275507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.293346] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1192.302460] LNet: Skipped 2 previous similar messages [ 1192.339630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.374076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1193.015334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_aborted -r 1 [ 1193.073766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 1 [ 1193.137949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_aborted -r 1 [ 1193.185742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 1 [ 1193.276715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_aborted -r 1 [ 1193.334885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 1 [ 1193.408120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_aborted -r 1 [ 1193.491960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 1 [ 1193.574809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1194.030535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.078858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1194.185713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1194.299344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1194.345177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1194.432860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1194.496771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1194.553601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1194.613892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1194.661764] LNet: 61043:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1194.687906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.716549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.747875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.789801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.827751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.874926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.938633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1195.538800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_no_route -r 1 [ 1195.597497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 1 [ 1195.671844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_no_route -r 1 [ 1195.725739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 1 [ 1195.788536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_no_route -r 1 [ 1195.882936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 1 [ 1195.934534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_no_route -r 1 [ 1195.983593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 1 [ 1196.090297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1196.141101] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1196.149620] LNet: Skipped 4 previous similar messages [ 1196.481553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.523731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1196.554812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.604207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.643267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1196.688598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.745216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.789786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.829700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.846534] LNet: 61167:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.872842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.933375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.978390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.015799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.055294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.110547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.178764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.834090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_timeout -r 1 [ 1197.889360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 1 [ 1197.929240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_timeout -r 1 [ 1198.012973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 1 [ 1198.094554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_timeout -r 1 [ 1198.133694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 1 [ 1198.175730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_timeout -r 1 [ 1198.222565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 1 [ 1198.262342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1198.508080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1198.542846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1198.587548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.622507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.668978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1198.720544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1198.777334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1198.826409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1198.879755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1198.974095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.021372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.085651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.127606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.157979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.197774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.238983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1199.856845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_error -r 1 [ 1199.919402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_error -r 1 [ 1199.956636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_error -r 1 [ 1199.984897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_error -r 1 [ 1200.019386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_error -r 1 [ 1200.058769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_error -r 1 [ 1200.093880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_error -r 1 [ 1200.134405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_error -r 1 [ 1200.174988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1200.196987] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1200.201796] LNet: Skipped 5 previous similar messages [ 1200.473893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.506664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1200.537651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1200.578316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1200.622198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1200.662206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1200.696602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1200.782700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1200.830619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1200.859373] LNet: 61414:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1200.864416] LNet: 61414:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1200.887486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1200.926395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1200.955492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.993545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.045271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.098296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.154345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1206.616750] LNet: 61490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1206.623619] LNetError: 61490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.630990] LNet: Removed LNI 192.168.203.45@tcp [ 1207.657228] LNet: Removed LNI 192.168.203.45@tcp1 [ 1208.214829] Key type .llcrypt unregistered [ 1208.218453] Key type ._llcrypt unregistered [ 1223.037866] Key type ._llcrypt registered [ 1223.044255] Key type .llcrypt registered [ 1224.332509] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:37:16 (1789576636) [ 1225.141516] Key type .llcrypt unregistered [ 1225.147505] Key type ._llcrypt unregistered [ 1230.572061] Key type ._llcrypt registered [ 1230.575206] Key type .llcrypt registered [ 1230.781479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1230.802785] alg: No test for adler32 (adler32-zlib) [ 1231.718267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1231.746341] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1238.805417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1239.559580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_dropped -r 1 [ 1239.612267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_dropped -r 1 [ 1239.670224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1239.690075] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1239.957633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.983648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.012918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.055588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.093622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1240.129334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.165959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.210732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.246640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.265641] LNet: 62736:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.291368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.336576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.393774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.445682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.475570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.509454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.552258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1241.156203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_error -r 1 [ 1241.195707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_error -r 1 [ 1241.231662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1241.248472] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1241.483878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.515649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.548421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.585391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.622969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1241.661378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.692443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.728665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.759125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.774916] LNet: 62848:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.796683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.830887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.860250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.887455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.922243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.962475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.004825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1242.556475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_timeout -r 1 [ 1242.605356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_timeout -r 1 [ 1242.651617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1242.679267] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1242.950165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.997411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.044448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.083943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.146739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1243.185980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.226535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.261213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.295878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.315383] LNet: 62959:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.332992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.375640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.404233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.430673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.461469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.493427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.524189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1243.798661] LNet: 63000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.804665] LNetError: 63000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.819320] LNet: Removed LNI 192.168.203.45@tcp [ 1244.246528] Key type .llcrypt unregistered [ 1244.253755] Key type ._llcrypt unregistered [ 1259.581511] Key type ._llcrypt registered [ 1259.585908] Key type .llcrypt registered [ 1261.278435] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:37:52 (1789576672) [ 1262.093693] Key type .llcrypt unregistered [ 1262.099669] Key type ._llcrypt unregistered [ 1269.028515] Key type ._llcrypt registered [ 1269.031489] Key type .llcrypt registered [ 1269.257445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1269.278919] alg: No test for adler32 (adler32-zlib) [ 1270.225362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1270.284268] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1277.666709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1286.114498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1286.158561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1286.210380] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1287.500380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_dropped -r 1 [ 1287.544523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 1 [ 1287.590801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_dropped -r 1 [ 1287.627275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 1 [ 1287.667934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_dropped -r 1 [ 1287.695802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 1 [ 1287.732977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_dropped -r 1 [ 1287.766054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 1 [ 1287.813868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1287.827273] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1287.832601] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 1288.027949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.060777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.092801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.125941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.164404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1288.234779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.296078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.331633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.378951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.400827] LNet: 64465:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.426823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.457460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.502293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.536918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.579675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.631369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.676673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1288.716786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1289.159371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_error -r 1 [ 1289.202524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_error -r 1 [ 1289.243372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_error -r 1 [ 1289.298536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_error -r 1 [ 1289.353786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_error -r 1 [ 1289.401575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_error -r 1 [ 1289.442665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_error -r 1 [ 1289.484543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_error -r 1 [ 1289.526470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1289.543712] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1289.552691] LNet: Skipped 2 previous similar messages [ 1289.756820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1289.795585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1289.828157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1289.852241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1289.884143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1289.922397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1289.962381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1290.010942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1290.053861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1290.077724] LNet: 64590:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1290.098490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.133307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.172636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.206905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.241826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.275556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1290.308129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1290.348530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.815217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_timeout -r 1 [ 1290.852414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_timeout -r 1 [ 1290.890652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_timeout -r 1 [ 1290.931591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_timeout -r 1 [ 1290.956756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_timeout -r 1 [ 1290.996918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_timeout -r 1 [ 1291.048605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_timeout -r 1 [ 1291.085394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_timeout -r 1 [ 1291.123910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.145@tcp [ 1291.146064] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1291.344439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1291.373489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1291.398934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1291.441528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1291.483476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1291.512220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1291.541177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1291.575659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1291.597850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1291.616571] LNet: 64715:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1291.637145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1291.678335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.717920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.754783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.797053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.825507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1291.859272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1291.903930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.488570] LNet: 64793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.497430] LNetError: 64793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.502909] LNet: Removed LNI 192.168.203.45@tcp [ 1296.519517] LNet: Removed LNI 192.168.203.45@tcp1 [ 1296.913658] Key type .llcrypt unregistered [ 1296.918341] Key type ._llcrypt unregistered [ 1312.068323] Key type ._llcrypt registered [ 1312.075174] Key type .llcrypt registered [ 1313.191891] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:38:45 (1789576725) [ 1314.949215] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1316.355880] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:38:48 (1789576728) [ 1317.054278] Key type .llcrypt unregistered [ 1317.060776] Key type ._llcrypt unregistered [ 1322.452325] Key type ._llcrypt registered [ 1322.454051] Key type .llcrypt registered [ 1322.647429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1322.662572] alg: No test for adler32 (adler32-zlib) [ 1323.572829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1323.595949] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1330.748950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1331.355637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e network_timeout -r 1 [ 1331.382969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e network_timeout -r 1 [ 1331.408731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1331.432202] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1331.649643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1331.668638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1331.702715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1331.732904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1331.758695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1331.792835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1331.808433] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1331.831633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1331.871460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1331.914736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1331.938363] LNet: 66336:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1331.960839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1332.000968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1332.052525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1332.101824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.145352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1332.182056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1332.435340] LNet: 66375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.440862] LNetError: 66375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.465183] LNet: Removed LNI 192.168.203.45@tcp [ 1332.831382] Key type .llcrypt unregistered [ 1332.834192] Key type ._llcrypt unregistered [ 1348.977408] Key type ._llcrypt registered [ 1348.981039] Key type .llcrypt registered [ 1349.126620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1349.143364] alg: No test for adler32 (adler32-zlib) [ 1350.044862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1350.063596] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1357.798479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1366.882560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.920295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1366.952862] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1369.160719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e network_timeout -r 1 [ 1369.236530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e network_timeout -r 1 [ 1369.292935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e network_timeout -r 1 [ 1369.355252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e network_timeout -r 1 [ 1369.414963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e network_timeout -r 1 [ 1369.492250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e network_timeout -r 1 [ 1369.531654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e network_timeout -r 1 [ 1369.607454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e network_timeout -r 1 [ 1369.665093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1369.695329] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1370.078679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1370.117176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1370.171320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1370.221302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1370.265989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1370.335845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1370.365256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1370.420906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1370.469309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1370.514775] LNet: 67253:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1370.527415] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1370.528324] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1370.575226] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 1370.585228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1370.652799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1370.712734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1370.763258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1370.850981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1370.921427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1375.165766] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1375.171796] LNet: Skipped 1 previous similar message [ 1375.347882] LNet: 67327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1375.354601] LNetError: 67327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1375.361337] LNet: Removed LNI 192.168.203.45@tcp [ 1375.372379] LNet: Removed LNI 192.168.203.45@tcp1 [ 1375.817493] Key type .llcrypt unregistered [ 1375.822914] Key type ._llcrypt unregistered [ 1389.022640] Key type ._llcrypt registered [ 1389.030842] Key type .llcrypt registered [ 1390.815704] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:40:02 (1789576802) [ 1391.172469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1391.203642] alg: No test for adler32 (adler32-zlib) [ 1392.165854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.259831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1392.301892] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1393.089963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1393.110820] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1394.593405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1394.778568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1394.880487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1394.965586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1395.015713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1395.106497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1395.218185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.45@tcp [ 1395.282121] LNetError: 68425:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1395.295483] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1395.298989] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1397.349825] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1398.078699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1401.440041] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1401.863418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1401.897979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1401.965536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1401.991502] LNet: 68523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1402.003661] LNetError: 68523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1402.008391] LNet: Removed LNI 192.168.203.45@tcp [ 1402.017885] LNet: Removed LNI 192.168.203.45@tcp1 [ 1402.036509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.102258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1402.123696] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1403.306090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1403.366642] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1404.908425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1404.972457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1405.085476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1405.146470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1405.199851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1405.252694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1405.306380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1405.342571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.45@tcp [ 1405.370652] LNetError: 68795:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1406.176762] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1409.057766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1412.319491] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1412.329294] LNet: Skipped 1 previous similar message [ 1417.249265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1417.317517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1417.361478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1424.164927] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:40:36 (1789576836) [ 1424.334095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1424.362878] LNet: 69514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1424.372755] LNetError: 69514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.377955] LNetError: 69514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1424.393936] LNet: Removed LNI 192.168.203.45@tcp [ 1424.409886] LNet: Removed LNI 192.168.203.45@tcp1 [ 1424.438326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.504671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1424.531625] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1425.238212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1425.272090] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1426.003205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1426.071351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1426.110385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1426.145812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1426.177711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1426.201394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1426.224329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.45@tcp --health 0 [ 1426.339498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1426.527166] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.45@tcp [ 1427.551854] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1427.560274] LNet: Skipped 1 previous similar message [ 1442.822160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1442.958524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1442.991940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1443.026509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.45@tcp --health 500 [ 1443.132523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1450.857755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1450.946877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.983824] LNet: 70022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.990671] LNetError: 70022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.999761] LNetError: 70022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1451.008482] LNet: Removed LNI 192.168.203.45@tcp [ 1451.021635] LNet: Removed LNI 192.168.203.45@tcp1 [ 1451.067114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.138931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1451.173413] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1451.903621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1451.923589] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1452.662024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1452.713345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1452.779887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1452.819369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1452.888138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1452.932670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1452.965847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1453.004699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.45@tcp --health 0 [ 1454.175797] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1454.189472] LNet: Skipped 3 previous similar messages [ 1454.316426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1464.620369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1464.661379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1464.706529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1472.168981] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:41:23 (1789576883) [ 1472.674648] LNet: 71068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.684954] LNetError: 71068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.688890] LNetError: 71068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1472.702366] LNet: Removed LNI 192.168.203.45@tcp [ 1472.744712] LNet: Removed LNI 192.168.203.45@tcp1 [ 1473.679713] Key type .llcrypt unregistered [ 1473.684174] Key type ._llcrypt unregistered [ 1480.101547] Key type ._llcrypt registered [ 1480.103427] Key type .llcrypt registered [ 1480.290778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1480.301254] alg: No test for adler32 (adler32-zlib) [ 1481.240970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1481.279383] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1489.953562] Lustre: DEBUG MARKER: Initial discovery [ 1490.019601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.145@tcp [ 1492.005174] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1492.111836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1492.147446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.145@tcp [ 1492.179668] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1492.208706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1492.244049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.145@tcp -r 1 -m GET -l 3 [ 1492.285192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.145@tcp -r 1 -m GET -e local_error [ 1492.320925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.145@tcp -r 1 -m PUT -l 6 [ 1492.575110] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1493.453631] Lustre: DEBUG MARKER: Force oleg345-client.virtnet to discover 192.168.203.145@tcp (in background) [ 1493.570153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.145@tcp [ 1494.649279] Lustre: DEBUG MARKER: Force oleg345-server to discover 192.168.203.45@tcp [ 1496.609880] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1497.030141] Lustre: DEBUG MARKER: Wait for 71825 [ 1498.395556] Lustre: DEBUG MARKER: Finished wait on 71825 [ 1498.527388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1498.567245] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1498.588686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1498.891424] LNet: 72278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.899218] LNetError: 72278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.942282] LNet: Removed LNI 192.168.203.45@tcp [ 1500.402691] Key type .llcrypt unregistered [ 1500.405258] Key type ._llcrypt unregistered [ 1515.854287] Key type ._llcrypt registered [ 1515.858879] Key type .llcrypt registered [ 1517.048950] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:42:08 (1789576928) [ 1517.532479] Key type .llcrypt unregistered [ 1517.538238] Key type ._llcrypt unregistered [ 1523.403664] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1523.633785] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1523.650193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1523.664408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1526.315500] Key type ._llcrypt registered [ 1526.318397] Key type .llcrypt registered [ 1526.484684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1526.501120] alg: No test for adler32 (adler32-zlib) [ 1527.405780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.482913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1527.527639] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1528.335988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1528.354875] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1537.363818] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:42:28 (1789576948) [ 1537.494420] LNet: 74344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1537.505692] LNetError: 74344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1537.519290] LNet: Removed LNI 10.1.2.3@tcp [ 1537.539819] LNet: Removed LNI 192.168.203.45@tcp [ 1538.279466] Key type .llcrypt unregistered [ 1538.284193] Key type ._llcrypt unregistered [ 1544.870646] Key type ._llcrypt registered [ 1544.879166] Key type .llcrypt registered [ 1545.213916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1545.238544] alg: No test for adler32 (adler32-zlib) [ 1546.194297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.265367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.302639] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1547.037922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1547.065774] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1548.078076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.45@tcp [ 1549.215633] LNetError: Unexpected error -22 connecting to 192.168.203.45@tcp at host 192.168.203.45:988 [ 1555.996892] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:42:47 (1789576967) [ 1556.365745] LNet: 75482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.374633] LNetError: 75482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1556.382541] LNet: Removed LNI 10.1.2.3@tcp [ 1556.417223] LNet: Removed LNI 192.168.203.45@tcp [ 1557.143145] Key type .llcrypt unregistered [ 1557.146500] Key type ._llcrypt unregistered [ 1564.617210] Key type ._llcrypt registered [ 1564.622731] Key type .llcrypt registered [ 1564.870906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1564.898391] alg: No test for adler32 (adler32-zlib) [ 1566.012453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.085987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1566.116274] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1567.145656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1567.184572] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 1568.226282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.45@tcp1 --nid 192.168.203.45@tcp2 [ 1573.352260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp1 192.168.203.45@tcp2 [ 1573.382318] LNetError: 76314:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.45@tcp1 to 192.168.203.45@tcp2 - Route table may be misconfigured [ 1573.412203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp2 192.168.203.45@tcp1 [ 1581.125554] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:43:12 (1789576992) [ 1581.226502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.258192] LNet: 76879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.267879] LNetError: 76879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1581.276296] LNet: Removed LNI 192.168.203.45@tcp1 [ 1582.313663] LNet: Removed LNI 192.168.203.45@tcp2 [ 1582.385883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.459399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.482938] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1583.388911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1583.418363] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1584.446217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 1584.504775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.45@tcp [ 1584.567692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp1 192.168.203.45@tcp1 [ 1584.632169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.45@tcp -d 192.168.203.45@tcp -e network_timeout [ 1584.676665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -e network_timeout [ 1584.712400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -e network_timeout [ 1584.751454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -e network_timeout [ 1584.783173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1584.797842] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1584.818704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1584.935565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1585.026330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1585.517804] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1591.630574] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:43:23 (1789577003) [ 1591.796315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.815705] LNet: 77745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.821839] LNetError: 77745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.836145] LNetError: 77745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1591.849302] LNet: Removed LNI 192.168.203.45@tcp [ 1592.871280] LNet: Removed LNI 192.168.203.45@tcp1 [ 1592.919816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.038845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1593.386427] LNet: 77787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.901755] Key type .llcrypt unregistered [ 1593.904443] Key type ._llcrypt unregistered [ 1604.126106] Key type ._llcrypt registered [ 1604.127882] Key type .llcrypt registered [ 1605.270508] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:43:37 (1789577017) [ 1605.726833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1605.737321] alg: No test for adler32 (adler32-zlib) [ 1606.676494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.105893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1609.694373] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:43:41 (1789577021) [ 1610.027565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.056465] LNet: 78882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.077810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.151484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1610.189818] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1610.935546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1610.960762] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1611.878785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1611.909618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp1 [ 1611.944175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp1 [ 1618.829190] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:43:50 (1789577030) [ 1620.808770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg345-server [ 1622.283815] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:43:54 (1789577034) [ 1623.763600] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg345-server [ 1625.422259] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:43:57 (1789577037) [ 1627.457150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg345-server [ 1629.453311] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:44:01 (1789577041) [ 1631.288472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg345-server [ 1632.760961] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:44:04 (1789577044) [ 1634.635451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg345-server [ 1636.367479] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:44:08 (1789577048) [ 1638.187945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg345-server [ 1639.926770] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:44:11 (1789577051) [ 1641.637132] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg345-server [ 1643.186828] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:44:15 (1789577055) [ 1645.135889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg345-server [ 1646.823640] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:44:18 (1789577058) [ 1648.288573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg345-server [ 1649.673964] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:44:21 (1789577061) [ 1649.931353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.965387] LNet: 82760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.969343] LNetError: 82760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1649.975649] LNet: Removed LNI 192.168.203.45@tcp [ 1651.050991] LNet: Removed LNI 192.168.203.45@tcp1 [ 1651.069406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.148588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1651.169691] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1651.779791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1651.853255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1652.017609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.043584] LNet: 82907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.050633] LNetError: 82907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.059276] LNetError: 82907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1652.082129] LNet: Removed LNI 192.168.203.45@tcp [ 1652.123171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.202530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1652.242312] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1652.919833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1654.242138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.257160] LNet: 83062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.261787] LNetError: 83062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.333994] LNet: Removed LNI 192.168.203.45@tcp [ 1655.386494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.478913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.530285] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1656.244908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1656.343644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1656.552378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.578589] LNetError: 83209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.603201] LNet: Removed LNI 192.168.203.45@tcp [ 1656.642123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.720266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1656.753485] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1657.411459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1657.640490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.730839] LNet: Removed LNI 192.168.203.45@tcp [ 1658.756601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.812558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.833320] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1659.599394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1659.722351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1659.948991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.968505] LNet: 83502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.975453] LNet: 83502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1661.030886] LNet: Removed LNI 192.168.203.45@tcp [ 1661.059782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.154443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1661.193162] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1661.861508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1662.064888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.086395] LNetError: 83647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.094978] LNetError: 83647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1662.115834] LNet: Removed LNI 192.168.203.45@tcp [ 1662.172367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.211925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1662.235358] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1662.943873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1663.015845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1663.311931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.422842] LNet: Removed LNI 192.168.203.45@tcp [ 1664.445861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.515088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1664.551854] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1665.291436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1665.547696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.601287] LNet: Removed LNI 192.168.203.45@tcp [ 1666.640404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.712336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1666.740398] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1667.569574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1667.646038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1667.885258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.966906] LNet: Removed LNI 192.168.203.45@tcp [ 1669.006507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.093299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1669.116208] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1670.012709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1670.291994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.325554] LNet: 84232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.335927] LNet: 84232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1670.343178] LNetError: 84232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.355260] LNetError: 84232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1670.374896] LNet: Removed LNI 192.168.203.45@tcp [ 1670.407216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.478863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1670.509174] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1671.221581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1671.281591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1671.520260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.623415] LNet: Removed LNI 192.168.203.45@tcp [ 1672.644666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.707947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1672.731558] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1673.458434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1673.757459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.855463] LNet: Removed LNI 192.168.203.45@tcp [ 1674.896609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.004109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1675.036903] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1675.970446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1676.045771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.080415] LNet: Removed LNI 192.168.203.45@tcp [ 1676.111620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.168944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1676.194409] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1676.983336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1677.058307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.090351] LNet: Removed LNI 192.168.203.45@tcp [ 1677.120613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.187865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1677.285881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.334567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.404693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1677.537306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.588324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.674051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1677.750979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.805314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.870521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1677.904397] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1678.978495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.007483] LNet: Removed LNI 192.168.203.45@tcp [ 1679.026341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.078324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1686.593760] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:44:58 (1789577098) [ 1686.728862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.763421] LNet: 85540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.769860] LNet: 85540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1686.828799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.924681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1686.983607] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1688.638969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.683900] LNetError: 85685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.693969] LNetError: 85685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1688.719047] LNet: Removed LNI 192.168.203.45@tcp231 [ 1688.777148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.886879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1688.954155] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1688.971435] LNet: 85693:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1689.836457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1689.896688] LNet: Removed LNI 192.168.203.45@tcp231 [ 1689.936260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1689.978386] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1691.463409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.500687] LNet: Removed LNI 192.168.203.45@tcp231 [ 1691.551980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.669986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1691.721278] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/0] [ 1691.728591] LNet: 85845:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1692.345848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1692.388699] LNet: Removed LNI 192.168.203.45@tcp231 [ 1692.424288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1692.455101] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/0] [ 1693.794456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.835440] LNet: Removed LNI 192.168.203.45@tcp231 [ 1693.873313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.967222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1694.010831] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/60] [ 1694.019916] LNet: 85997:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1694.520685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1694.550858] LNet: Removed LNI 192.168.203.45@tcp231 [ 1694.575585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1694.602928] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/60] [ 1695.785870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.842774] LNet: Removed LNI 192.168.203.45@tcp231 [ 1695.869461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.938230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1695.966882] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1696.536774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1696.589416] LNet: Removed LNI 192.168.203.45@tcp231 [ 1696.616872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1696.650919] LNet: Added LNI 192.168.203.45@tcp231 [8/256/0/180] [ 1704.353844] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:45:16 (1789577116) [ 1704.480541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.518604] LNet: Removed LNI 192.168.203.45@tcp231 [ 1704.561085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.632072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1704.669118] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1705.237260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1705.275890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1705.294142] LNet: Removed LNI 192.168.203.45@tcp232 [ 1705.320434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1705.357940] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1706.087010] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1706.136376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1706.168438] LNet: Removed LNI 192.168.203.45@tcp232 [ 1706.188869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1706.225225] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1707.023345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1707.075753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1707.125490] LNet: Removed LNI 192.168.203.45@tcp232 [ 1707.150580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1707.183706] LNet: Added LNI 192.168.203.45@tcp232 [8/256/0/180] [ 1707.995753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1708.039685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1708.076549] LNet: Removed LNI 192.168.203.45@tcp232 [ 1708.104921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1714.098736] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:45:26 (1789577126) [ 1715.121853] Key type .llcrypt unregistered [ 1715.124684] Key type ._llcrypt unregistered [ 1721.676762] Key type ._llcrypt registered [ 1721.680842] Key type .llcrypt registered [ 1721.859445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1721.879282] alg: No test for adler32 (adler32-zlib) [ 1722.789904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1722.813157] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1729.907413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1739.150144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.259289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1739.309619] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1741.412658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_interrupt -r 2 [ 1741.476342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 2 [ 1741.541110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_interrupt -r 2 [ 1741.588043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 2 [ 1741.625801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_interrupt -r 2 [ 1741.671090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_interrupt -r 2 [ 1741.732634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_interrupt -r 2 [ 1741.778884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_interrupt -r 2 [ 1741.824646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1741.846750] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1741.852665] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1742.152165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1742.192845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1742.251123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1742.295755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1742.329838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1742.383951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1742.432072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1742.474548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1742.503553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1742.520778] LNet: 88650:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1742.539279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1742.572184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1742.607896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1742.647540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1742.684566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1742.712579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1743.287414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_dropped -r 2 [ 1743.324785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 2 [ 1743.363568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_dropped -r 2 [ 1743.402892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 2 [ 1743.439361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_dropped -r 2 [ 1743.473862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_dropped -r 2 [ 1743.509451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_dropped -r 2 [ 1743.541731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_dropped -r 2 [ 1743.592783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1743.620693] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1743.901699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1743.942485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1743.971284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1744.005956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1744.059093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1744.098345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1744.147859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1744.193352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1744.229417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1744.252484] LNet: 88773:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1744.274888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1744.299167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1744.319513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1744.359729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1744.406868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1744.440280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1744.926723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_aborted -r 2 [ 1744.963688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 2 [ 1745.024345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_aborted -r 2 [ 1745.074297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 2 [ 1745.131738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_aborted -r 2 [ 1745.161920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_aborted -r 2 [ 1745.195921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_aborted -r 2 [ 1745.231233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_aborted -r 2 [ 1745.263772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1745.283590] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1745.572625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1745.609531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1745.645680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1745.687655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1745.716171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1745.745117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1745.775903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1745.810995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1745.854284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1745.872682] LNet: 88895:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1745.889581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1745.913538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1745.934547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1745.961568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1745.988401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1746.031695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1746.455695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_no_route -r 2 [ 1746.505698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 2 [ 1746.536797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_no_route -r 2 [ 1746.595655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 2 [ 1746.644516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_no_route -r 2 [ 1746.695717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_no_route -r 2 [ 1746.753551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_no_route -r 2 [ 1746.799948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_no_route -r 2 [ 1746.828674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1747.076197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1747.107756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1747.130295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1747.150591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1747.175232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1747.199400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1747.225717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1747.254714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1747.295853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1747.338508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1747.393049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1747.422986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1747.445107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1747.466522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1747.502241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1747.990474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e local_timeout -r 2 [ 1748.028238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 2 [ 1748.068517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e local_timeout -r 2 [ 1748.102293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 2 [ 1748.135591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e local_timeout -r 2 [ 1748.178610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e local_timeout -r 2 [ 1748.215076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e local_timeout -r 2 [ 1748.244714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e local_timeout -r 2 [ 1748.275433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1748.302695] LNet: There was an unexpected network error while writing to 192.168.203.145: rc = -22 [ 1748.310187] LNet: Skipped 1 previous similar message [ 1748.571373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1748.633817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1748.665840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1748.710031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1748.784051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1748.826674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1748.866559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1748.909644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1748.939802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1748.960569] LNet: 89140:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1748.970121] LNet: 89140:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1748.997638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1749.020984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1749.049534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1749.094191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1749.146501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1749.174867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1749.751914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp -m GET -e remote_dropped -r 2 [ 1749.798832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 2 [ 1749.841645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp -m GET -e remote_dropped -r 2 [ 1749.872705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 2 [ 1749.923948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp -m GET -e remote_dropped -r 2 [ 1749.969665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.45@tcp1 -d 192.168.203.145@tcp1 -m GET -e remote_dropped -r 2 [ 1750.023927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp -m GET -e remote_dropped -r 2 [ 1750.073779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.45@tcp1 -d 192.168.203.45@tcp1 -m GET -e remote_dropped -r 2 [ 1750.120251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.45@tcp 192.168.203.145@tcp [ 1750.154922] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 1750.399294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1750.432971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1750.466390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1750.506877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1750.547665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1750.588366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1750.632667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1750.669793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1750.719691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1750.769822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1750.804732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1750.855566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1750.891983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1750.930833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1750.971755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1756.498263] LNet: 89336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1756.508639] LNetError: 89336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1756.514651] LNet: Removed LNI 192.168.203.45@tcp [ 1756.529903] LNet: Removed LNI 192.168.203.45@tcp1 [ 1756.995112] Key type .llcrypt unregistered [ 1757.000818] Key type ._llcrypt unregistered [ 1774.300297] Key type ._llcrypt registered [ 1774.303161] Key type .llcrypt registered [ 1775.763816] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:46:27 (1789577187) [ 1776.191784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1776.204394] alg: No test for adler32 (adler32-zlib) [ 1777.175322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.237654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.259929] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1778.231516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1778.263458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.45@tcp [ 1778.281409] LNetError: 90312:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1778.362570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.45@tcp [ 1778.456342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1785.806733] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:46:37 (1789577197) [ 1786.154949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.193847] LNet: 90902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.203107] LNetError: 90902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1786.226221] LNet: Removed LNI 192.168.203.45@tcp [ 1786.261753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.353808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1786.400488] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1787.375614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1787.398578] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 1788.541760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.45@tcp [ 1788.582772] LNetError: 91169:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.45@tcp to recovery [ 1789.408180] LNet: 1 local NIs in recovery (showing 1): 192.168.203.45@tcp [ 1789.410367] LNet: There was an unexpected network error while writing to 192.168.203.45: rc = -22 [ 1789.864833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.45@tcp [ 1799.525720] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:46:51 (1789577211) [ 1801.671153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg345-server [ 1804.139528] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:46:55 (1789577215) [ 1806.913809] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg345-server [ 1808.489426] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:47:00 (1789577220) [ 1810.274248] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg345-server [ 1811.900565] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:47:03 (1789577223) [ 1812.347223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.385866] LNet: 92799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.391639] LNetError: 92799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.409657] LNet: Removed LNI 192.168.203.45@tcp [ 1812.433187] LNet: Removed LNI 192.168.203.45@tcp1 [ 1812.490692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.587038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.612094] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1813.446851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1813.572483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1813.635161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1813.795319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1813.828801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1813.928487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1820.153296] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:47:12 (1789577232) [ 1820.417975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.483264] LNet: 93531:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.491752] LNetError: 93531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.500959] LNetError: 93531:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1820.518833] LNet: Removed LNI 192.168.203.45@tcp [ 1820.548972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.657315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1820.684987] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1828.708386] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:47:20 (1789577240) [ 1828.924616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.965183] LNet: 94233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.969911] LNetError: 94233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.990112] LNet: Removed LNI 192.168.203.45@tcp [ 1829.038495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.150204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1829.218257] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1836.432098] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:47:28 (1789577248) [ 1838.057151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1839.834126] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:47:31 (1789577251) [ 1840.297227] LNet: 95270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.302302] LNetError: 95270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.313452] LNet: Removed LNI 192.168.203.45@tcp [ 1840.965908] Key type .llcrypt unregistered [ 1840.971814] Key type ._llcrypt unregistered [ 1846.976851] Key type ._llcrypt registered [ 1846.978438] Key type .llcrypt registered [ 1847.225259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1847.251586] alg: No test for adler32 (adler32-zlib) [ 1848.199748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1848.230974] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1855.396811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 1859.598125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.145@tcp [ 1859.962918] LNet: 95684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.971787] LNetError: 95684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1859.986851] LNet: Removed LNI 192.168.203.45@tcp [ 1860.513863] Key type .llcrypt unregistered [ 1860.516554] Key type ._llcrypt unregistered [ 1872.395342] Key type ._llcrypt registered [ 1872.397812] Key type .llcrypt registered [ 1873.987858] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1875.640271] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1877.345753] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:48:09 (1789577289) [ 1878.871971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1880.838722] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:48:12 (1789577292) [ 1882.853217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg345-server [ 1885.033605] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:48:16 (1789577296) [ 1887.173690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg345-server [ 1889.458574] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:48:20 (1789577300) [ 1891.114870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1891.345452] alg: No test for adler32 (adler32-zlib) [ 1892.590559] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 1892.964650] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1894.714495] Key type lgssc registered [ 1896.085533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1916.306649] Key type lgssc unregistered [ 1916.571271] LNet: 98856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.578638] LNetError: 98856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.602849] LNet: Removed LNI 192.168.203.45@tcp [ 1917.189809] Key type .llcrypt unregistered [ 1917.196417] Key type ._llcrypt unregistered [ 1931.085218] Key type ._llcrypt registered [ 1931.091658] Key type .llcrypt registered [ 1932.588681] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:49:04 (1789577344) [ 1933.125552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1933.160823] alg: No test for adler32 (adler32-zlib) [ 1934.092546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1934.185042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1934.241761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1934.273284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1934.310897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1941.647695] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:49:13 (1789577353) [ 1941.753562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1941.787724] LNet: 100264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.823410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.908167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.932986] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1942.769822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1942.793950] LNet: 100414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1942.800263] LNetError: 100414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.816154] LNet: Removed LNI 192.168.203.45@tcp [ 1942.864947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1942.939798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1942.994082] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1943.632371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.665737] LNetError: 100443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.690929] LNet: Removed LNI 192.168.203.45@tcp [ 1943.715396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.773258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1943.795462] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1950.856869] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:49:22 (1789577362) [ 1951.004328] LNet: 101029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.012918] LNet: 101029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1951.025645] LNetError: 101029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.062110] LNet: Removed LNI 192.168.203.45@tcp [ 1951.737915] Key type .llcrypt unregistered [ 1951.740093] Key type ._llcrypt unregistered [ 1957.238226] Key type ._llcrypt registered [ 1957.247137] Key type .llcrypt registered [ 1957.474625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1957.485617] alg: No test for adler32 (adler32-zlib) [ 1958.355600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.424747] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 1958.429718] LNet: *** cfs_fail_loc=e003, val=100*** [ 1959.177234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 1959.202507] LNet: *** cfs_fail_loc=e003, val=100*** [ 1959.207441] LNet: Skipped 2 previous similar messages [ 1959.216424] Ping reply received with mapping: [ 1959.220562] found metadata reply NI[192.168.203.45@tcp] [ 1959.223674] 2d cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1959.228507] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1959.232402] 5a 5a 5a 5a 5a 5a 5a 5a [ 1959.369795] LNet: 101428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.383689] LNetError: 101428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.415884] LNet: Removed LNI 192.168.203.45@tcp [ 1960.137489] Key type .llcrypt unregistered [ 1960.141569] Key type ._llcrypt unregistered [ 1971.213229] Key type ._llcrypt registered [ 1971.215754] Key type .llcrypt registered [ 1972.706813] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:49:44 (1789577384) [ 1973.266416] Key type .llcrypt unregistered [ 1973.269505] Key type ._llcrypt unregistered [ 1980.183311] Key type ._llcrypt registered [ 1980.189132] Key type .llcrypt registered [ 1980.621809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1980.640491] alg: No test for adler32 (adler32-zlib) [ 1981.957227] LNetError: 102465:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1982.497695] Key type .llcrypt unregistered [ 1982.502706] Key type ._llcrypt unregistered [ 1987.931089] Key type ._llcrypt registered [ 1987.933658] Key type .llcrypt registered [ 1989.333699] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:50:01 (1789577401) [ 1989.753393] Key type .llcrypt unregistered [ 1989.759404] Key type ._llcrypt unregistered [ 1994.447822] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1994.452641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1996.935636] Key type ._llcrypt registered [ 1996.939200] Key type .llcrypt registered [ 1997.169309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1997.187637] alg: No test for adler32 (adler32-zlib) [ 1998.162272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1998.262697] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2007.797802] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:50:19 (1789577419) [ 2007.935604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.965994] LNet: 104234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.972771] LNetError: 104234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2008.008279] LNet: Removed LNI 10.1.2.3@tcp [ 2008.054519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.202815] Key type .llcrypt unregistered [ 2009.210589] Key type ._llcrypt unregistered [ 2015.330709] Key type ._llcrypt registered [ 2015.342201] Key type .llcrypt registered [ 2015.588266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2015.611637] alg: No test for adler32 (adler32-zlib) [ 2016.632433] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2016.719523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2016.920789] LNet: 104526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.930402] LNetError: 104526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.950685] LNet: Removed LNI 192.168.203.45@tcp [ 2017.559609] Key type .llcrypt unregistered [ 2017.563534] Key type ._llcrypt unregistered [ 2029.918227] Key type ._llcrypt registered [ 2029.923623] Key type .llcrypt registered [ 2031.259460] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:50:43 (1789577443) [ 2032.177408] Key type .llcrypt unregistered [ 2032.181705] Key type ._llcrypt unregistered [ 2039.709283] Key type ._llcrypt registered [ 2039.711811] Key type .llcrypt registered [ 2039.966446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2039.996613] alg: No test for adler32 (adler32-zlib) [ 2046.640304] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:50:58 (1789577458) [ 2046.800046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2046.852514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.921682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2046.955552] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2047.672636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2047.702515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2047.728285] LNet: Removed LNI 192.168.203.45@tcp [ 2047.914850] LNet: 106264:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.267715] Key type .llcrypt unregistered [ 2048.271307] Key type ._llcrypt unregistered [ 2059.098863] Key type ._llcrypt registered [ 2059.102321] Key type .llcrypt registered [ 2060.438646] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:51:12 (1789577472) [ 2060.776261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2060.799816] alg: No test for adler32 (adler32-zlib) [ 2061.747375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.796186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2061.815645] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2062.535348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2062.571382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.45@tcp [ 2062.595236] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2062.604578] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2062.612871] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2062, 0, 0 [ 2062.629813] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2062.637467] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2062.646905] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2062.652537] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2062, 0, 0 [ 2062.657839] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2062.660342] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2062.663365] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2062.669497] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2062, 0, 2112 [ 2062.677482] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2062.682415] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2062.693516] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2062.703391] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2062, 0, 2112 [ 2062.709348] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2067.892497] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:51:19 (1789577479) [ 2068.302337] LNet: 107801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.310985] LNetError: 107801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2068.336940] LNet: Removed LNI 192.168.203.45@tcp [ 2068.947658] Key type .llcrypt unregistered [ 2068.950076] Key type ._llcrypt unregistered [ 2074.282217] Key type ._llcrypt registered [ 2074.285293] Key type .llcrypt registered [ 2074.420505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2074.431297] alg: No test for adler32 (adler32-zlib) [ 2075.302920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2075.333811] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2082.395344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 2091.158690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.223147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2091.264365] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 2092.092341] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2092.235783] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2092.249242] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2092.258280] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2094.657166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2094.683382] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2105.916400] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:51:57 (1789577517) [ 2106.343232] LNet: 109304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.348706] LNetError: 109304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.357636] LNet: Removed LNI 192.168.203.45@tcp [ 2107.423326] LNet: Removed LNI 192.168.203.45@tcp1 [ 2107.428828] LNetError: 109304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2107.443812] LNetError: 109304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2107.470802] LNet: Removed LNI 10.1.2.3@tcp99 [ 2108.445765] Key type .llcrypt unregistered [ 2108.450091] Key type ._llcrypt unregistered [ 2114.786314] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2114.791337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2117.547569] Key type ._llcrypt registered [ 2117.555857] Key type .llcrypt registered [ 2117.784779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2117.823723] alg: No test for adler32 (adler32-zlib) [ 2118.823444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.927418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2118.994418] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2119.984549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2120.020042] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2122.393424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2122.450236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2122.534619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2122.567953] LNetError: 110048:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2122.612846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2122.691910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2123.313466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2123.399498] LNet: 110068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.404681] LNetError: 110068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2123.419575] LNet: Removed LNI 10.1.2.3@tcp [ 2124.473657] LNet: Removed LNI 192.168.203.45@tcp [ 2125.226509] Key type .llcrypt unregistered [ 2125.232091] Key type ._llcrypt unregistered [ 2136.170297] Key type ._llcrypt registered [ 2136.175462] Key type .llcrypt registered [ 2137.655971] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:52:29 (1789577549) [ 2138.212739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2138.252948] alg: No test for adler32 (adler32-zlib) [ 2139.161604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.202989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.228170] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2139.925673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2139.984670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg345-client@tcp [ 2145.899614] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:52:37 (1789577557) [ 2146.045067] LNet: 111607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2146.066089] LNetError: 111607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2147.126431] LNet: Removed LNI 192.168.203.45@tcp [ 2147.970743] Key type .llcrypt unregistered [ 2147.974799] Key type ._llcrypt unregistered [ 2153.757374] Key type ._llcrypt registered [ 2153.761219] Key type .llcrypt registered [ 2153.977338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2153.998749] alg: No test for adler32 (adler32-zlib) [ 2154.900448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2154.934920] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2154.977117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2161.595666] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:52:53 (1789577573) [ 2163.376431] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2165.173516] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:52:57 (1789577577) [ 2166.361136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2167.814777] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:52:59 (1789577579) [ 2168.069678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.099244] LNet: 113035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.115347] LNetError: 113035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.134338] LNet: Removed LNI 192.168.203.45@tcp [ 2168.151459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2168.203269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2168.233769] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2168.908384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2168.967317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2175.679185] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:53:07 (1789577587) [ 2175.761169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.810377] LNet: 113739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2175.822807] LNetError: 113739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.842814] LNet: Removed LNI 192.168.203.45@tcp [ 2175.881680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.930482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2175.975374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2176.615553] Key type .llcrypt unregistered [ 2176.616928] Key type ._llcrypt unregistered [ 2186.954467] Key type ._llcrypt registered [ 2186.958853] Key type .llcrypt registered [ 2188.513421] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:53:20 (1789577600) [ 2188.941488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2188.957522] alg: No test for adler32 (adler32-zlib) [ 2189.853195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.922997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2189.969232] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2190.675917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2190.740420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2196.831532] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:53:28 (1789577608) [ 2197.026296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2197.040911] LNet: 115298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2197.046579] LNetError: 115298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.059630] LNet: Removed LNI 192.168.203.45@tcp [ 2198.092661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2198.172812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2198.198262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2204.133690] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:53:36 (1789577616) [ 2204.260465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2204.278563] LNet: 115872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.298866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2204.396660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2204.794413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2204.836848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2204.903503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2205.361228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2205.407589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2205.468341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.488974] LNet: 115912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.507622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.571784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2206.102360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2206.132653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2206.214359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2206.685882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2206.733292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2214.626246] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:53:46 (1789577626) [ 2214.732789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.777467] LNet: 116504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.834382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.957355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2215.007959] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2216.101353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2216.135306] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2217.318665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2217.368423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2217.589379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.611214] LNetError: 116792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.619292] LNet: Removed LNI 192.168.203.45@tcp [ 2217.634067] LNet: Removed LNI 192.168.203.45@tcp2 [ 2217.654680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2217.691930] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2217.696042] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2217.700358] LNet: 116798:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2218.286428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2218.460513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2219.487166] LNet: Removed LNI 192.168.203.45@tcp [ 2219.490094] LNetError: 116835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2219.507586] LNetError: 116835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2219.531896] LNet: Removed LNI 192.168.203.45@tcp2 [ 2219.587989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.679330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2219.705144] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2220.515852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2220.545077] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2221.619609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2221.682424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2221.761705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2222.024724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.064372] LNet: 117120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.070501] LNet: 117120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2222.081970] LNetError: 117120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.090142] LNet: Removed LNI 192.168.203.45@tcp [ 2223.142611] LNet: Removed LNI 192.168.203.45@tcp2 [ 2223.233909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2223.345601] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2223.357177] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2223.364707] LNet: 117126:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2224.131470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2224.458635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.499541] LNet: Removed LNI 192.168.203.45@tcp [ 2225.576418] LNet: Removed LNI 192.168.203.45@tcp2 [ 2225.642976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.731936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2225.768852] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2226.579054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2226.613732] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2227.567275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2227.628744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2227.815853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.835364] LNetError: 117443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.840567] LNetError: 117443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2227.847132] LNet: Removed LNI 192.168.203.45@tcp [ 2227.864211] LNet: Removed LNI 192.168.203.45@tcp2 [ 2227.906263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.965040] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2227.970860] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2227.974247] LNet: 117449:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2228.510948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2228.656703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2229.727138] LNet: Removed LNI 192.168.203.45@tcp [ 2229.737854] LNet: Removed LNI 192.168.203.45@tcp2 [ 2229.786718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2229.849840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2229.876928] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2230.587870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2230.604863] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2231.286540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2231.314217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2231.386190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2231.615494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.633542] LNet: 117771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.645147] LNet: 117771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2232.671129] LNet: Removed LNI 192.168.203.45@tcp [ 2233.703953] LNet: Removed LNI 192.168.203.45@tcp2 [ 2233.755775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.854477] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2233.859252] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2233.864088] LNet: 117777:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2234.424706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2234.637148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.671255] LNet: Removed LNI 192.168.203.45@tcp [ 2235.752223] LNet: Removed LNI 192.168.203.45@tcp2 [ 2235.798459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.878905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.913084] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2236.676621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2236.697843] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2237.521670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2237.549908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2237.617489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2237.972763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2238.003096] LNetError: 118111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.007664] LNetError: 118111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2238.012871] LNet: Removed LNI 192.168.203.45@tcp [ 2238.023645] LNet: Removed LNI 192.168.203.45@tcp2 [ 2238.061750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.151994] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2238.163146] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2238.167324] LNet: 118117:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2238.663882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2239.041592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2240.095267] LNet: Removed LNI 192.168.203.45@tcp [ 2240.107714] LNet: Removed LNI 192.168.203.45@tcp2 [ 2240.139306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.209991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2240.237022] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2240.862623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2240.881776] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2241.800172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2241.847987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2241.899755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2241.959217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2242.302740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2243.359233] LNet: Removed LNI 192.168.203.45@tcp [ 2244.395625] LNet: Removed LNI 192.168.203.45@tcp2 [ 2244.421307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.507906] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2244.515613] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2245.040289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2251.864433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2253.761341] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:54:25 (1789577665) [ 2253.878953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2253.899479] LNet: 119232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.904237] LNet: 119232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2253.911720] LNet: Removed LNI 192.168.203.45@tcp [ 2254.950961] LNet: Removed LNI 192.168.203.45@tcp2 [ 2254.983978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2255.456213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2255.491915] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2256.277484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2256.319558] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2257.668376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2257.699885] LNetError: 119511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2257.704895] LNetError: 119511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2257.710351] LNet: Removed LNI 10.1.2.3@tcp [ 2257.726580] LNet: Removed LNI 192.168.203.45@tcp [ 2257.756739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2257.817149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2257.860322] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2257.872356] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2258.388398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2258.422859] LNet: Removed LNI 192.168.203.45@tcp [ 2258.432893] LNet: Removed LNI 10.1.2.3@tcp [ 2265.400517] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:54:37 (1789577677) [ 2265.719512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.802067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.855995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2265.890835] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2266.779900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2266.820160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2266.871615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2266.921782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2266.958326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2266.990884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2267.051063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2267.103701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2267.163165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2267.222365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2267.303369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2267.339846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2267.498361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2273.985961] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 12:54:45 (1789577685) [ 2274.087716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2275.177694] LNet: Removed LNI 192.168.203.45@tcp [ 2275.215947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.282814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2275.309451] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2276.301900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2276.342854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2276.387482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2276.464844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2276.511281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2276.580755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2276.648723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2276.681584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2276.728181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2276.776917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2276.821500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2276.885659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2277.040882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2277.102057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2277.149811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2277.191440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2277.264843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2277.308890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2277.351743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2277.392507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2277.445697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2277.484942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2277.529891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.45@tcp [ 2284.593914] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:54:56 (1789577696) [ 2284.714800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2285.797819] LNet: Removed LNI 192.168.203.45@tcp [ 2285.840920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.951797] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2286.185157] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2286.190221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2286.198522] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2288.686951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2288.731797] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2289.927258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2289.956448] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2292.249962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2292.324937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2292.400507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.45@tcp -m PUT -e local_timeout -r 1 [ 2292.451861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2292.515414] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2292.520817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2292.775834] LNetError: 121674:0:(lib-move.c:4794:lnet_parse()) 192.168.203.45@tcp, src 192.168.203.45@tcp: Dropping PUT (error -108 looking up sender) [ 2293.792478] LNet: 122133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.807697] LNet: 122133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2293.818951] LNetError: 122133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2293.827975] LNetError: 122133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2293.837465] LNet: Removed LNI 10.1.2.3@tcp [ 2294.892469] LNet: Removed LNI 192.168.203.45@tcp [ 2295.514011] Key type .llcrypt unregistered [ 2295.517639] Key type ._llcrypt unregistered [ 2307.851637] Key type ._llcrypt registered [ 2307.854519] Key type .llcrypt registered [ 2309.791974] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:55:20 (1789577720) [ 2310.198577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2310.233644] alg: No test for adler32 (adler32-zlib) [ 2311.190616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.543590] LNet: 122984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.187483] Key type .llcrypt unregistered [ 2312.191040] Key type ._llcrypt unregistered [ 2319.482618] Key type ._llcrypt registered [ 2319.486295] Key type .llcrypt registered [ 2319.700804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2319.720590] alg: No test for adler32 (adler32-zlib) [ 2320.673661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2320.711115] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2329.122620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 2329.443847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2330.115542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.145@tcp [ 2330.447346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2330.492525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2330.524067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2330.568985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2330.617805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2330.653090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2330.701564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2330.758169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2330.797739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2330.811290] LNet: 123434:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2330.825251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2330.859142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2330.898020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2330.945407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2330.967969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2330.980587] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 2331.019180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2331.682485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2331.920777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2331.957801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2332.018181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2332.045918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2332.086718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2332.141605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2332.199650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2332.232192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2332.268303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2332.298361] LNet: 123554:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2332.316909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2332.346847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2332.381482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2332.429781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2332.474800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2332.508371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2333.180525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.145@tcp [ 2333.449887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2333.497764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2333.539176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2333.566289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2333.586785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2333.616646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2333.654411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2333.689455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2333.727286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2333.745901] LNet: 123673:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2333.764162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2333.794555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2333.825522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2333.876296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2333.903250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2333.926504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2334.560792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2334.804360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2334.829872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2334.858262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2334.882382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2334.909261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2334.938444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2334.966288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2335.001193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2335.047176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2335.099433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2335.157717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2335.224376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2335.272250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2335.333200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2335.395658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2335.758635] LNet: 123838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.769928] LNetError: 123838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.803934] LNet: Removed LNI 192.168.203.45@tcp [ 2336.257811] Key type .llcrypt unregistered [ 2336.264864] Key type ._llcrypt unregistered [ 2354.144183] Key type ._llcrypt registered [ 2354.151105] Key type .llcrypt registered [ 2355.500849] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:56:07 (1789577767) [ 2356.088327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2356.116473] alg: No test for adler32 (adler32-zlib) [ 2357.075386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.496629] LNet: 124724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2358.281362] Key type .llcrypt unregistered [ 2358.285222] Key type ._llcrypt unregistered [ 2365.595896] Key type ._llcrypt registered [ 2365.602591] Key type .llcrypt registered [ 2365.829640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2365.852351] alg: No test for adler32 (adler32-zlib) [ 2366.798807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2366.840551] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2375.007655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.145@tcp [ 2384.940637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2384.997465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2385.033278] LNet: Added LNI 192.168.203.45@tcp1 [8/256/0/180] [ 2386.164649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2386.783988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.145@tcp [ 2387.050688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2387.100286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2387.149878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2387.206083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2387.243805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2387.277852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2387.319920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2387.355772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2387.360659] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.145@tcp [ 2387.390965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2387.411971] LNet: 125377:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2387.428518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2387.462258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2387.498487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2387.562691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2387.601676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2387.650307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2388.491267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2388.810975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2388.863464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2388.902447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2388.943893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2388.978465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2389.011342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2389.044370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2389.078849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2389.105913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2389.123729] LNet: 125497:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2389.136722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2389.180125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2389.209835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2389.238571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2389.265708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2389.314414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2390.113075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.145@tcp [ 2390.421357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2390.460503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2390.501168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2390.543305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2390.585041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2390.626456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2390.670975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2390.717696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2390.793121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2390.846613] LNet: 125616:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2390.883301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2390.945333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2391.007186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2391.059062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2391.129449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2391.186326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2392.002071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2392.310602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2392.354250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2392.399330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2392.456115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2392.493392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2392.523617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2392.561665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2392.601300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2392.649455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2392.695855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2392.722634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2392.768941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2392.817269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2392.876819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2392.941653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2397.569450] LNet: 125816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.579684] LNetError: 125816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.591176] LNet: Removed LNI 192.168.203.45@tcp [ 2397.598054] LNet: Removed LNI 192.168.203.45@tcp1 [ 2397.958581] Key type .llcrypt unregistered [ 2397.962687] Key type ._llcrypt unregistered [ 2413.158902] Key type ._llcrypt registered [ 2413.161963] Key type .llcrypt registered [ 2414.321401] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:57:06 (1789577826) [ 2414.694900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2414.705951] alg: No test for adler32 (adler32-zlib) [ 2415.641480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.699725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2415.719584] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2416.399107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2416.432708] LNet: Added LNI 192.168.203.45@tcp2 [8/256/0/180] [ 2417.257759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.45@tcp [ 2417.335942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2418.400116] LNet: Removed LNI 192.168.203.45@tcp [ 2424.545988] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:57:16 (1789577836) [ 2425.882720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2428.054226] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:57:19 (1789577839) [ 2435.656329] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:57:27 (1789577847) [ 2443.094423] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:57:34 (1789577854) [ 2450.176940] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:57:42 (1789577862) [ 2457.639659] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:57:49 (1789577869) [ 2464.359803] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:57:56 (1789577876) [ 2470.925681] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:58:02 (1789577882) [ 2478.107344] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:58:09 (1789577889) [ 2484.756627] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:58:16 (1789577896) [ 2491.410657] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:58:23 (1789577903) [ 2497.324657] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:58:29 (1789577909) [ 2503.678877] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:58:35 (1789577915) [ 2510.243751] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:58:42 (1789577922) [ 2517.444440] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:58:49 (1789577929) [ 2524.145384] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:58:56 (1789577936) [ 2531.180644] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:59:02 (1789577942) [ 2539.369989] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:59:10 (1789577950) [ 2548.895230] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:59:20 (1789577960) [ 2549.036361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2549.074176] LNet: 137501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2549.084657] LNetError: 137501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2549.104361] LNet: Removed LNI 192.168.203.45@tcp2 [ 2549.163769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2549.400359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2549.429620] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2550.890138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2550.949607] LNet: 137649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2550.968889] LNetError: 137649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2550.984227] LNet: Removed LNI 192.168.203.45@tcp [ 2551.030142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2551.121459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2551.171524] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2559.122523] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:59:30 (1789577970) [ 2566.907816] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:59:38 (1789577978) [ 2574.896308] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:59:46 (1789577986) [ 2583.179942] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:59:55 (1789577995) [ 2590.274358] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 13:00:02 (1789578002) [ 2598.064189] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 13:00:09 (1789578009) [ 2606.326492] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 13:00:17 (1789578017) [ 2615.343387] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 13:00:26 (1789578026) [ 2623.789622] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 13:00:35 (1789578035) [ 2632.000323] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 13:00:43 (1789578043) [ 2639.556066] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 13:00:51 (1789578051) [ 2647.478771] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 13:00:59 (1789578059) [ 2654.965150] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2396 sec ========== 13:01:06 (1789578066) [ 2655.359488] LNet: 145129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2655.371375] LNetError: 145129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2655.414732] LNet: Removed LNI 192.168.203.45@tcp [ 2656.181746] Key type .llcrypt unregistered [ 2656.185038] Key type ._llcrypt unregistered [ 2669.121477] Key type ._llcrypt registered [ 2669.129778] Key type .llcrypt registered [ 2669.407148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2669.428665] alg: No test for adler32 (adler32-zlib) [ 2670.536324] Lustre: Lustre: Build Version: 2.17.58_39_gf14a6ef [ 2670.782165] LNet: Added LNI 192.168.203.45@tcp [8/256/0/180] [ 2672.543271] Key type lgssc registered [ 2674.142067] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2739.349936] Lustre: Mounted lustre-client - version 2.17.58_39_gf14a6ef [ 2744.238459] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2752.661025] Lustre: Unmounted lustre-client [ 2780.659526] Key type lgssc unregistered [ 2780.930776] LNet: 148077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2780.944811] LNetError: 148077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2780.963624] LNet: Removed LNI 192.168.203.45@tcp [ 2781.735240] Key type .llcrypt unregistered [ 2781.739752] Key type ._llcrypt unregistered