[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 442498929 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002446] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009134] LSM: Security Framework initializing [ 0.010079] Yama: becoming mindful. [ 0.011038] SELinux: Initializing. [ 0.012064] *** VALIDATE selinux *** [ 0.020816] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025342] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027138] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] *** VALIDATE tmpfs *** [ 0.031037] *** VALIDATE proc *** [ 0.032225] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034415] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041382] debug: unmapping init [mem 0xffffffffb3459000-0xffffffffb3460fff] [ 0.044151] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045791] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048014] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053313] rcu: Hierarchical SRCU implementation. [ 0.055352] smp: Bringing up secondary CPUs ... [ 0.056552] x86: Booting SMP configuration: [ 0.057030] .... node #0, CPUs: #1 #2 #3 [ 0.061347] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.147217] node 0 deferred pages initialised in 81ms [ 0.151079] devtmpfs: initialized [ 0.152270] x86/mm: Memory block size: 128MB [ 0.155848] gcov: version magic: 0x41383552 [ 0.158298] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162113] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166356] pinctrl core: initialized pinctrl subsystem [ 0.168190] [ 0.168843] ************************************************************* [ 0.172015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175011] ** ** [ 0.177013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.180014] ** ** [ 0.183012] ** This means that this kernel is built to expose internal ** [ 0.185012] ** IOMMU data structures, which may compromise security on ** [ 0.188013] ** your system. ** [ 0.190011] ** ** [ 0.192010] ** If you see this message and you are not debugging the ** [ 0.194013] ** kernel, report this immediately to your vendor! ** [ 0.197014] ** ** [ 0.199011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.202015] ************************************************************* [ 0.204573] NET: Registered protocol family 16 [ 0.207415] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.209052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.212056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.215096] cpuidle: using governor menu [ 0.216761] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217335] PCI: Using configuration type 1 for base access [ 0.220136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.231076] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.234037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.239659] cryptd: max_cpu_qlen set to 1000 [ 0.242143] ACPI: Added _OSI(Module Device) [ 0.244012] ACPI: Added _OSI(Processor Device) [ 0.246013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.252567] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.257577] ACPI: Interpreter enabled [ 0.259066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.261012] ACPI: Using IOAPIC for interrupt routing [ 0.263147] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266393] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276743] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.279052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.282019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.285080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.291352] acpiphp: Slot [2] registered [ 0.292143] acpiphp: Slot [5] registered [ 0.294104] acpiphp: Slot [6] registered [ 0.295132] acpiphp: Slot [3] registered [ 0.297175] acpiphp: Slot [4] registered [ 0.299145] acpiphp: Slot [7] registered [ 0.300093] acpiphp: Slot [8] registered [ 0.302101] acpiphp: Slot [9] registered [ 0.304129] acpiphp: Slot [10] registered [ 0.306157] acpiphp: Slot [11] registered [ 0.308142] acpiphp: Slot [12] registered [ 0.310087] acpiphp: Slot [13] registered [ 0.312142] acpiphp: Slot [14] registered [ 0.314162] acpiphp: Slot [15] registered [ 0.316123] acpiphp: Slot [16] registered [ 0.317101] acpiphp: Slot [17] registered [ 0.319105] acpiphp: Slot [18] registered [ 0.321225] acpiphp: Slot [19] registered [ 0.323135] acpiphp: Slot [20] registered [ 0.325107] acpiphp: Slot [21] registered [ 0.326074] acpiphp: Slot [22] registered [ 0.328108] acpiphp: Slot [23] registered [ 0.330122] acpiphp: Slot [24] registered [ 0.332118] acpiphp: Slot [25] registered [ 0.333174] acpiphp: Slot [26] registered [ 0.335143] acpiphp: Slot [27] registered [ 0.337106] acpiphp: Slot [28] registered [ 0.338114] acpiphp: Slot [29] registered [ 0.340092] acpiphp: Slot [30] registered [ 0.341113] acpiphp: Slot [31] registered [ 0.342043] PCI host bridge to bus 0000:00 [ 0.344019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.346038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.348033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.352029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.355024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.359028] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.361178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.365757] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.367809] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.374617] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.379658] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.384037] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.387020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.390019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.393411] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.396858] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.399038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.402831] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.406922] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.414011] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.418845] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.422732] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.426838] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.432785] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.448012] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.457211] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.463015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.471791] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.485897] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.495801] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.498438] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.501433] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.504430] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.507260] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.512672] iommu: Default domain type: Passthrough [ 0.514387] SCSI subsystem initialized [ 0.516161] ACPI: bus type USB registered [ 0.517090] usbcore: registered new interface driver usbfs [ 0.519081] usbcore: registered new interface driver hub [ 0.521077] usbcore: registered new device driver usb [ 0.523166] pps_core: LinuxPPS API ver. 1 registered [ 0.524008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.527057] PTP clock support registered [ 0.529037] EDAC MC: Ver: 3.0.0 [ 0.530359] PCI: Using ACPI for IRQ routing [ 0.532714] NetLabel: Initializing [ 0.534011] NetLabel: domain hash size = 128 [ 0.536010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.538093] NetLabel: unlabeled traffic allowed by default [ 0.541039] vgaarb: loaded [ 0.542267] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.544013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.550343] clocksource: Switched to clocksource kvm-clock [ 0.664734] VFS: Disk quotas dquot_6.6.0 [ 0.666554] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.669307] *** VALIDATE ramfs *** [ 0.670673] *** VALIDATE hugetlbfs *** [ 0.673168] pnp: PnP ACPI init [ 0.675873] pnp: PnP ACPI: found 6 devices [ 0.693540] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.697335] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.699845] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.702332] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.705221] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.707788] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.711026] NET: Registered protocol family 2 [ 0.714054] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.719395] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.723568] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.729269] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.733858] TCP: Hash tables configured (established 65536 bind 65536) [ 0.737185] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.740385] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.743663] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.746773] NET: Registered protocol family 1 [ 0.749362] RPC: Registered named UNIX socket transport module. [ 0.752273] RPC: Registered udp transport module. [ 0.754624] RPC: Registered tcp transport module. [ 0.756825] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.759688] NET: Registered protocol family 44 [ 0.761533] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.764235] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.766754] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.769474] PCI: CLS 0 bytes, default 64 [ 0.771452] Unpacking initramfs... [ 2.266854] debug: unmapping init [mem 0xffff8e7ebcc64000-0xffff8e7ebffcffff] [ 2.271492] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.275158] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.278737] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.780553] Initialise system trusted keyrings [ 2.782633] Key type blacklist registered [ 2.784823] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.794560] zbud: loaded [ 2.797545] *** VALIDATE nfs *** [ 2.798923] *** VALIDATE nfs4 *** [ 2.801052] pstore: using deflate compression [ 2.805705] Platform Keyring initialized [ 2.953645] NET: Registered protocol family 38 [ 2.955696] Key type asymmetric registered [ 2.957663] Asymmetric key parser 'x509' registered [ 2.960147] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.963264] io scheduler mq-deadline registered [ 2.966066] io scheduler kyber registered [ 2.967985] io scheduler bfq registered [ 2.970130] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.974309] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.976971] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.980304] ACPI: Power Button [PWRF] [ 2.987291] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.994221] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.005879] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.035345] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.065426] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.071080] Non-volatile memory driver v1.3 [ 3.072779] Linux agpgart interface v0.103 [ 3.112978] virtio_blk virtio1: [vda] 132376 512-byte logical blocks (67.8 MB/64.6 MiB) [ 3.117609] vda: detected capacity change from 0 to 67776512 [ 3.137697] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.141693] vdb: detected capacity change from 0 to 1073741824 [ 3.150115] libphy: Fixed MDIO Bus: probed [ 3.155280] usbcore: registered new interface driver usbserial_generic [ 3.157419] usbserial: USB Serial support registered for generic [ 3.160130] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.165603] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.167398] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.171419] mousedev: PS/2 mouse device common for all mice [ 3.174610] rtc_cmos 00:05: RTC can wake from S4 [ 3.177582] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.183241] rtc_cmos 00:05: registered as rtc0 [ 3.185686] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.186539] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.194223] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.194235] intel_pstate: CPU model not supported [ 3.201151] hid: raw HID events driver (C) Jiri Kosina [ 3.203430] usbcore: registered new interface driver usbhid [ 3.205035] usbhid: USB HID core driver [ 3.206806] drop_monitor: Initializing network drop monitor service [ 3.209090] Initializing XFRM netlink socket [ 3.210664] NET: Registered protocol family 10 [ 3.214314] Segment Routing with IPv6 [ 3.215956] NET: Registered protocol family 17 [ 3.218192] mpls_gso: MPLS GSO support [ 3.224788] RAS: Correctable Errors collector initialized. [ 3.227968] AVX version of gcm_enc/dec engaged. [ 3.229584] AES CTR mode by8 optimization enabled [ 3.306937] sched_clock: Marking stable (3306911530, 0)->(4200763487, -893851957) [ 3.310511] registered taskstats version 1 [ 3.312597] Loading compiled-in X.509 certificates [ 3.314984] zswap: loaded using pool lzo/zbud [ 3.342390] Key type big_key registered [ 3.354914] Key type encrypted registered [ 3.356431] ima: No TPM chip found, activating TPM-bypass! [ 3.358696] ima: Allocated hash algorithm: sha1 [ 3.361112] ima: No architecture policies found [ 3.363527] evm: Initialising EVM extended attributes: [ 3.366363] evm: security.selinux [ 3.368012] evm: security.ima [ 3.369373] evm: security.capability [ 3.370917] evm: HMAC attrs: 0x1 [ 3.373650] rtc_cmos 00:05: setting system clock to 2026-03-30 12:40:04 UTC (1774874404) [ 3.381575] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 3.384907] debug: unmapping init [mem 0xffffffffb3182000-0xffffffffb3458fff] [ 3.393104] Write protecting the kernel read-only data: 28672k [ 3.396973] debug: unmapping init [mem 0xffffffffb1803000-0xffffffffb19fffff] [ 3.400064] debug: unmapping init [mem 0xffffffffb2114000-0xffffffffb21fffff] [ 3.457519] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.468349] systemd[1]: Detected virtualization kvm. [ 3.470525] systemd[1]: Detected architecture x86-64. [ 3.472441] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.500223] systemd[1]: No hostname configured. [ 3.501639] systemd[1]: Set hostname to . [ 3.504818] random: systemd: uninitialized urandom read (16 bytes read) [ 3.506809] systemd[1]: Initializing machine ID from random generator. [ 3.634896] random: systemd: uninitialized urandom read (16 bytes read) [ 3.638094] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.643728] random: systemd: uninitialized urandom read (16 bytes read) [ 3.646436] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.651834] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.276378] device-mapper: uevent: version 1.0.3 [ 4.278643] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Co[ 4.684941] random: fast init done ldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.033917] virtio_net virtio0 ens2: renamed from eth0 [ 5.107084] scsi host0: ata_piix [ 5.188574] scsi host1: ata_piix [ 5.190295] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.193116] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.823770] dracut-initqueue[584]: RTNETLINK answers: File exists [ 9.875828] random: crng init done [ 9.877564] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.305967] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.529992] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.802903] SELinux: Disabled at runtime. [ 11.864313] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.873340] systemd[1]: Detected virtualization kvm. [ 11.875259] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.354029] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.358336] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.364405] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.368942] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.372762] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.381452] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.389462] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.503923] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.896118] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.260944] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.299590] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.379467] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.406301] EDAC sbridge: Ver: 1.1.2 [ 14.685592] Key type dns_resolver registered [ 14.991765] NFS: Registering the id_resolver key type [ 14.993673] Key type id_resolver registered [ 14.994861] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg609-client login: [ 52.701879] libcfs: loading out-of-tree module taints kernel. [ 52.745453] Key type ._llcrypt registered [ 52.748661] Key type .llcrypt registered [ 53.452093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 53.468231] alg: No test for adler32 (adler32-zlib) [ 54.842341] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 55.720854] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 57.479265] Key type lgssc registered [ 59.026334] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.043013] hrtimer: interrupt took 1823449 ns [ 158.773559] Lustre: Mounted lustre-client [ 163.291030] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 177.285148] Lustre: DEBUG MARKER: oleg609-client.virtnet: executing check_logdir /tmp/testlogs/ [ 181.469389] Lustre: DEBUG MARKER: oleg609-client.virtnet: executing yml_node [ 184.288199] Lustre: lustre-OST0000-osc-ffff8e7f0963f800: disconnect after 23s idle [ 185.065000] Lustre: DEBUG MARKER: Client: 2.17.51.23 [ 187.045705] Lustre: DEBUG MARKER: MDS: 2.17.51.23 [ 189.110246] Lustre: DEBUG MARKER: OSS: 2.17.51.23 [ 190.309618] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Mar 30 08:43:10 EDT 2026 [ 200.955179] LustreError: 7900:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e7f0963f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 200.991667] Lustre: Unmounted lustre-client [ 235.770284] Key type lgssc unregistered [ 236.076371] LNet: 8350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 236.089249] LNetError: 8350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 236.110383] LNet: Removed LNI 192.168.206.9@tcp [ 236.900224] Key type .llcrypt unregistered [ 236.903912] Key type ._llcrypt unregistered [ 255.156213] Key type ._llcrypt registered [ 255.159437] Key type .llcrypt registered [ 255.334051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 255.355693] alg: No test for adler32 (adler32-zlib) [ 256.235956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 256.299872] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 256.323956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 259.447262] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 259.555743] LNet: 9225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.570194] LNetError: 9225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 259.585974] LNet: Removed LNI 192.168.206.9@tcp [ 260.066694] Key type .llcrypt unregistered [ 260.071181] Key type ._llcrypt unregistered [ 264.707605] Key type ._llcrypt registered [ 264.709424] Key type .llcrypt registered [ 265.837700] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:44:25 (1774874665) [ 266.099671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 266.108835] alg: No test for adler32 (adler32-zlib) [ 266.990311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.061266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.078452] LNet: 9639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 267.095550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.137764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 267.160951] LNet: 9646:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 272.040709] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:44:32 (1774874672) [ 272.253137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.267817] LNet: 10223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 272.286322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.343124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 277.757943] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:44:37 (1774874677) [ 277.831704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 277.847589] LNet: 10802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 277.868641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 277.954861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 283.324876] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:44:43 (1774874683) [ 283.402853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.427858] LNet: 11385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.451782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.533778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 288.637212] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:44:48 (1774874688) [ 288.747960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.776846] LNet: 11973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.814221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.877532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 289.372806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 294.462204] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:44:54 (1774874694) [ 294.567883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.621082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.689500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 294.994273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 295.384101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 300.919941] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:45:00 (1774874700) [ 300.986568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.018933] LNet: 13155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.024213] LNet: 13155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 301.039963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.122408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 308.340350] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:45:08 (1774874708) [ 308.430352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.485738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.549080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 308.571717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 308.905897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 308.935792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 309.186589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 309.209767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 309.481553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 309.506642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 309.772256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 309.800285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 310.089699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 310.115036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 310.354627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 310.383198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 310.793657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 310.822454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 311.295748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 311.337297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 311.800872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 311.847480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 312.229400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 312.258050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 312.687184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 312.726485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 313.092995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 313.137904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 318.533037] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:45:18 (1774874718) [ 318.601965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.626177] LNet: 14422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.631390] LNet: 14422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 318.653272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.738638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 318.775879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 324.242264] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:45:24 (1774874724) [ 324.314990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.353432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.442888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 324.484889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 329.606360] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:45:29 (1774874729) [ 329.657646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.705251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.774918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 329.812452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 334.683341] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:45:34 (1774874734) [ 334.835582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.873922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.934810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 334.967286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 339.734509] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:45:39 (1774874739) [ 339.895959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.928477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 340.010331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 340.050907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 345.897105] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:45:45 (1774874745) [ 345.991061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.046896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.163171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 346.227217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 352.609431] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:45:52 (1774874752) [ 352.707517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.746740] LNet: 17939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.753538] LNet: 17939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 352.785116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.927639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 353.541722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 360.875304] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:46:00 (1774874760) [ 360.959351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.007563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.224287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 361.525634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 368.974401] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:46:08 (1774874768) [ 369.213975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.267861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.397889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 369.661948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 375.914083] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:46:15 (1774874775) [ 375.968486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.013048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.404920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 376.649715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 382.231958] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:46:22 (1774874782) [ 382.348817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.410754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.510193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 382.591194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 387.939991] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:46:27 (1774874787) [ 388.028228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.063311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.121892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 388.181672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 394.909398] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:46:34 (1774874794) [ 395.035406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.116500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.252389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 401.430490] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:46:41 (1774874801) [ 401.489639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.555584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.660631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 407.588628] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:46:47 (1774874807) [ 407.665648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.697489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.796331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 408.231475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 414.297424] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:46:53 (1774874813) [ 414.440196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.497076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.620603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 414.659802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 421.058714] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:47:00 (1774874820) [ 421.229209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.250314] LNet: 23972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.260580] LNet: 23972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 421.297536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.373448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 421.413548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 427.025787] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:47:07 (1774874827) [ 427.187285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.234668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.298888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 427.333907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 433.545865] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:47:13 (1774874833) [ 433.634406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.696483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.801354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 433.842597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 439.372989] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:47:19 (1774874839) [ 439.475589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.527285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.577511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 439.621739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 439.636508] LNetError: 25746:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 439.671800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 439.697235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 445.338519] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:47:25 (1774874845) [ 445.450453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.475843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.558545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LzmnT14uQu [ 445.632060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LzmnT14uQu [ 445.674323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LzmnT14uQu [ 450.086742] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:47:30 (1774874850) [ 450.287493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.325152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.378477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 450.423902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 450.488181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 455.771424] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:47:35 (1774874855) [ 455.843650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.912482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.999385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 458.614554] LNet: Message forwarding will be enabled in 2s [ 461.727144] LNet: Message forwarding enabled [ 462.153587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 462.177471] LNet: Message forwarding will stop in 2s to 6s [ 466.682477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 467.332731] LNet: Message forwarding disabled [ 468.455258] LNet: Message forwarding will be enabled in 2s [ 472.479217] LNet: Message forwarding enabled [ 474.423603] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:47:54 (1774874874) [ 474.545873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.557399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.650717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 480.582324] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:48:00 (1774874880) [ 480.717061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.770635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.814767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 480.835165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 480.865932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 480.890889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 480.920803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 480.957892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 481.003464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 481.047924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 481.087977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 481.147355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 481.197867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 481.233343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 481.276160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 481.308506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 481.333820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 481.363382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 481.392074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 481.429321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 481.457985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 481.483155] LNetError: 28815:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 486.295822] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:48:06 (1774874886) [ 486.420260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.459647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.509814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.489676] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:48:11 (1774874891) [ 492.828041] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg609-server [ 494.088226] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:48:14 (1774874894) [ 494.299212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.362488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.406629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 494.432245] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 495.676662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[10-14]@tcp [ 495.692948] LNet: 30530:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 495.728753] LNetError: 30320:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.10@tcp is being used as a gateway but routing feature is not turned on [ 495.740468] LNetError: 30320:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.10@tcp has gone from up to down [ 496.750468] LNetError: 30320:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.14@tcp is being used as a gateway but routing feature is not turned on [ 496.762420] LNetError: 30320:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 497.765525] LNetError: 30320:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.13@tcp is being used as a gateway but routing feature is not turned on [ 497.783594] LNetError: 30320:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 499.811652] LNetError: 30320:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.13@tcp is being used as a gateway but routing feature is not turned on [ 499.823482] LNetError: 30320:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 500.585266] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:48:20 (1774874900) [ 501.956135] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg609-server [ 503.201984] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:48:23 (1774874903) [ 503.373374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.391063] LNetError: 31464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 503.412526] LNet: Removed LNI 192.168.206.9@tcp [ 503.441568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.489902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 503.510813] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 504.004821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[10-15/2]@tcp [ 504.022566] LNet: 31581:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 504.030403] LNet: 31581:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 504.050221] LNetError: 31468:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.12@tcp is being used as a gateway but routing feature is not turned on [ 504.050880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[10-15/2]@tcp [ 504.079525] LNetError: 31468:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 504.093782] LNetError: 31468:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.12@tcp has gone from up to down [ 504.103509] LNetError: 31468:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 508.719332] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:48:28 (1774874908) [ 508.809560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.828936] LNetError: 32170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 508.841939] LNet: Removed LNI 192.168.206.9@tcp [ 508.870529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.939580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 508.987788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.019292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.060764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 509.100672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.124304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.158896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 509.213534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.239428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.282603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 509.330975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.354366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.393874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 509.441763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.470238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.523347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 509.579155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.603657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.637975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 509.686771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.714957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.775910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 509.834497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.867863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.924370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 509.983941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.013375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.072174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 510.135876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.169843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.200553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 510.238357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.269498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.321583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 514.415223] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:48:34 (1774874914) [ 515.601038] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg609-server [ 516.758814] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:48:36 (1774874916) [ 517.917559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg609-server [ 519.094536] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:48:39 (1774874919) [ 519.586463] Key type .llcrypt unregistered [ 519.588724] Key type ._llcrypt unregistered [ 523.310444] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 523.416965] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 523.424573] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 523.427747] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 525.848389] Key type ._llcrypt registered [ 525.851691] Key type .llcrypt registered [ 525.979538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 525.991059] alg: No test for adler32 (adler32-zlib) [ 526.850359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.889749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 526.905346] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 527.366040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 527.389320] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 529.101461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 529.118414] LNet: Removed LNI 10.1.2.3@tcp [ 529.243508] LNet: 34290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 529.247641] LNetError: 34290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 529.260176] LNet: Removed LNI 192.168.206.9@tcp [ 529.748507] Key type .llcrypt unregistered [ 529.749910] Key type ._llcrypt unregistered [ 536.959210] Key type ._llcrypt registered [ 536.961191] Key type .llcrypt registered [ 537.908887] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:48:57 (1774874937) [ 538.195733] Key type .llcrypt unregistered [ 538.201082] Key type ._llcrypt unregistered [ 541.899890] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 542.038487] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 542.045670] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 542.053792] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 544.392946] Key type ._llcrypt registered [ 544.396686] Key type .llcrypt registered [ 544.516188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 544.528767] alg: No test for adler32 (adler32-zlib) [ 545.387942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.424267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 545.446491] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 545.913391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 545.927794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 547.466614] LNet: 35781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.472743] LNetError: 35781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 547.483496] LNet: Removed LNI 192.168.206.9@tcp [ 547.497116] LNet: Removed LNI 10.1.2.3@tcp [ 547.917619] Key type .llcrypt unregistered [ 547.922772] Key type ._llcrypt unregistered [ 554.077875] Key type ._llcrypt registered [ 554.080500] Key type .llcrypt registered [ 554.842406] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:49:15 (1774874955) [ 555.132745] Key type .llcrypt unregistered [ 555.136106] Key type ._llcrypt unregistered [ 558.632694] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 558.699621] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 558.704182] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 558.720399] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 561.146923] Key type ._llcrypt registered [ 561.149473] Key type .llcrypt registered [ 561.332805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 561.355460] alg: No test for adler32 (adler32-zlib) [ 562.267242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.318866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 562.336770] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 563.943711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 563.966387] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 565.307582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 565.326398] LNet: Removed LNI 10.1.2.3@tcp [ 565.347634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 565.386619] LNet: Removed LNI 10.1.2.31@tcp [ 565.404608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 565.429185] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 566.068992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 566.096164] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 566.744315] LNet: 37697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.750475] LNetError: 37697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 566.756301] LNet: Removed LNI 10.1.2.3@tcp [ 566.768069] LNet: Removed LNI 10.1.2.31@tcp [ 567.189678] Key type .llcrypt unregistered [ 567.192103] Key type ._llcrypt unregistered [ 573.852978] Key type ._llcrypt registered [ 573.855246] Key type .llcrypt registered [ 574.772277] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:49:34 (1774874974) [ 575.016260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 575.035094] alg: No test for adler32 (adler32-zlib) [ 575.940218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.984114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 576.004863] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 576.626682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.642512] LNet: 38647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.649348] LNetError: 38647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.661780] LNet: Removed LNI 192.168.206.9@tcp [ 576.674952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.727933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.9@tcp [ 576.756692] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 577.273680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.9@tcp [ 577.298676] LNet: Removed LNI 192.168.206.9@tcp [ 577.350193] LNet: 38782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.745499] Key type .llcrypt unregistered [ 577.747304] Key type ._llcrypt unregistered [ 584.848454] Key type ._llcrypt registered [ 584.851766] Key type .llcrypt registered [ 585.816362] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:49:45 (1774874985) [ 587.114738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg609-server [ 588.420073] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:49:48 (1774874988) [ 588.811867] Key type .llcrypt unregistered [ 588.816986] Key type ._llcrypt unregistered [ 592.603943] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 592.700617] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 592.711200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 592.716949] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 595.203692] Key type ._llcrypt registered [ 595.206630] Key type .llcrypt registered [ 595.345225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 595.359608] alg: No test for adler32 (adler32-zlib) [ 596.257806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.313896] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 596.877926] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 598.527904] LNet: 40615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 598.535443] LNetError: 40615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 598.542405] LNet: Removed LNI 192.168.206.9@tcp1 [ 598.557898] LNet: Removed LNI 10.1.2.3@tcp2 [ 599.003352] Key type .llcrypt unregistered [ 599.004914] Key type ._llcrypt unregistered [ 607.226779] Key type ._llcrypt registered [ 607.228662] Key type .llcrypt registered [ 608.181902] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:50:08 (1774875008) [ 608.533696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 608.557342] alg: No test for adler32 (adler32-zlib) [ 609.407035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.451509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 609.467496] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 610.111389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.126902] LNet: 41583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.131596] LNetError: 41583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 610.144482] LNet: Removed LNI 192.168.206.9@tcp [ 610.158506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.194965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 610.223043] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 614.857399] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 08:50:14 (1774875014) [ 614.934421] LNet: 42165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.941951] LNetError: 42165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.954716] LNet: Removed LNI 192.168.206.9@tcp [ 615.423389] Key type .llcrypt unregistered [ 615.427562] Key type ._llcrypt unregistered [ 619.047229] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 619.140527] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 619.143585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 619.148056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 621.525819] Key type ._llcrypt registered [ 621.528201] Key type .llcrypt registered [ 621.636734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 621.649402] alg: No test for adler32 (adler32-zlib) [ 622.496269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.541339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 622.564276] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 623.045538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 623.056958] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 624.397638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.414667] LNet: 42910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.419259] LNetError: 42910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 624.423524] LNet: Removed LNI 192.168.206.9@tcp [ 624.438935] LNet: Removed LNI 10.1.2.3@tcp [ 624.460934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.514405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 624.538834] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 624.544195] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 624.850193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.879721] LNet: Removed LNI 192.168.206.9@tcp [ 624.891446] LNet: Removed LNI 10.1.2.3@tcp [ 624.923590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.975399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 624.995158] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 624.998139] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 625.223856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 625.238688] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 625.819804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.836917] LNet: 43099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 625.842943] LNet: 43099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 625.848724] LNetError: 43099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 625.852871] LNetError: 43099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 625.856819] LNet: Removed LNI 192.168.206.9@tcp [ 625.859068] LNet: Removed LNI 10.1.2.3@tcp [ 625.868180] LNet: Removed LNI 192.168.206.9@tcp2 [ 625.880932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.917930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 625.934449] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 625.938027] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 625.942025] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 630.312177] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 08:50:30 (1774875030) [ 630.414130] LNet: 43695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.420677] LNetError: 43695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.427038] LNetError: 43695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 630.434241] LNet: Removed LNI 192.168.206.9@tcp [ 630.439909] LNet: Removed LNI 10.1.2.3@tcp [ 630.452890] LNet: Removed LNI 192.168.206.9@tcp2 [ 630.796237] Key type .llcrypt unregistered [ 630.802244] Key type ._llcrypt unregistered [ 634.236847] Key type ._llcrypt registered [ 634.238177] Key type .llcrypt registered [ 634.363805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 634.377473] alg: No test for adler32 (adler32-zlib) [ 635.230215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.270137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.292784] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 635.879335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.893286] LNet: 44090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.899505] LNetError: 44090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.909963] LNet: Removed LNI 192.168.206.9@tcp [ 635.921280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.961767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 635.977531] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 636.262234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.286839] LNet: Removed LNI 192.168.206.9@tcp [ 636.299735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.340984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 636.362164] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 636.607185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.619042] LNet: 44150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.622828] LNet: 44150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 636.627628] LNetError: 44150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.632703] LNetError: 44150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 636.643926] LNet: Removed LNI 192.168.206.9@tcp [ 636.667784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.729750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 636.759093] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 637.037725] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 637.097514] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 637.101854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 637.106508] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 639.317092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.327501] LNet: 44224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.331596] LNetError: 44224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.349699] LNet: Removed LNI 192.168.206.9@tcp [ 639.371566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.426868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 639.468065] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 643.617588] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 08:50:43 (1774875043) [ 643.729431] LNet: 44820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.737863] LNetError: 44820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.758391] LNet: Removed LNI 192.168.206.9@tcp [ 644.128666] Key type .llcrypt unregistered [ 644.130569] Key type ._llcrypt unregistered [ 647.756345] Key type ._llcrypt registered [ 647.758166] Key type .llcrypt registered [ 647.859491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 647.869419] alg: No test for adler32 (adler32-zlib) [ 648.703426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.741732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.758385] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 649.293086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.308402] LNet: 45220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.315884] LNetError: 45220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.330611] LNet: Removed LNI 192.168.206.9@tcp [ 649.347594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.385542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 649.408280] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 653.768502] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 08:50:53 (1774875053) [ 653.984390] LNet: 45811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.991433] LNetError: 45811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.008529] LNet: Removed LNI 192.168.206.9@tcp [ 654.501851] Key type .llcrypt unregistered [ 654.505918] Key type ._llcrypt unregistered [ 658.282537] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 658.389252] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 658.391770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 658.394233] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 660.749925] Key type ._llcrypt registered [ 660.752128] Key type .llcrypt registered [ 660.882915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 660.891748] alg: No test for adler32 (adler32-zlib) [ 661.751283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.794487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.829657] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 662.469464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.485323] LNet: 46262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.489954] LNetError: 46262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 662.504554] LNet: Removed LNI 192.168.206.9@tcp [ 662.523555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.575585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 662.602484] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 667.717640] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 08:51:07 (1774875067) [ 667.902792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.918630] LNet: 46859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.926820] LNetError: 46859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 667.942068] LNet: Removed LNI 192.168.206.9@tcp [ 667.961128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.039308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 672.885820] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 08:51:12 (1774875072) [ 672.976607] LNet: 47434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.399621] Key type .llcrypt unregistered [ 673.401410] Key type ._llcrypt unregistered [ 677.772835] Key type ._llcrypt registered [ 677.776798] Key type .llcrypt registered [ 677.934334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 677.948862] alg: No test for adler32 (adler32-zlib) [ 678.835604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.891626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 678.910818] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 679.497695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.508817] LNet: 47840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.512376] LNetError: 47840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.528461] LNet: Removed LNI 192.168.206.9@tcp [ 679.546485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.604507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 679.626951] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 685.001028] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 08:51:24 (1774875084) [ 685.119687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.149163] LNet: 48426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.155755] LNetError: 48426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.180028] LNet: Removed LNI 192.168.206.9@tcp [ 685.202602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.287458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 690.456981] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 08:51:30 (1774875090) [ 691.600630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 693.112874] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 08:51:33 (1774875093) [ 694.636088] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 696.146162] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 08:51:36 (1774875096) [ 697.427710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 698.887197] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 08:51:38 (1774875098) [ 699.953947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 701.251337] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 08:51:41 (1774875101) [ 702.407850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 703.677891] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 08:51:43 (1774875103) [ 703.934881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.956984] LNet: 50484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.985591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.022971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.039108] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 704.570950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.9 [ 704.585832] LNet: 50617:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 704.591924] LNet: Removed LNI 192.168.206.9@tcp [ 710.339051] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 08:51:50 (1774875110) [ 710.445741] LNet: 51180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.455941] LNetError: 51180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.475487] LNet: Removed LNI 192.168.206.9@tcp [ 711.158686] Key type .llcrypt unregistered [ 711.160494] Key type ._llcrypt unregistered [ 716.532692] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 716.792532] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 716.799646] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 716.806292] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 718.488331] Key type ._llcrypt registered [ 718.489990] Key type .llcrypt registered [ 718.639395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 718.656971] alg: No test for adler32 (adler32-zlib) [ 720.180900] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 726.350492] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 08:52:06 (1774875126) [ 726.647942] LNet: 52086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.659228] LNetError: 52086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 726.681374] LNet: Removed LNI 10.1.2.3@tcp [ 727.227078] Key type .llcrypt unregistered [ 727.230120] Key type ._llcrypt unregistered [ 732.301500] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 732.499482] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 732.506655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 732.509980] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 735.102420] Key type ._llcrypt registered [ 735.105490] Key type .llcrypt registered [ 735.245768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 735.264491] alg: No test for adler32 (adler32-zlib) [ 736.178304] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 741.191687] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 08:52:21 (1774875141) [ 741.432940] LNet: 52979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.438761] LNetError: 52979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 741.466352] LNet: Removed LNI 10.1.2.3@tcp [ 741.966939] Key type .llcrypt unregistered [ 741.970363] Key type ._llcrypt unregistered [ 747.331698] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 747.530625] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 747.534520] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 747.539406] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 750.137244] Key type ._llcrypt registered [ 750.139391] Key type .llcrypt registered [ 750.266147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 750.292393] alg: No test for adler32 (adler32-zlib) [ 751.229406] LNet: 53279:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 751.250059] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 756.656092] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 08:52:36 (1774875156) [ 756.711069] LNet: 53862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.721342] LNetError: 53862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 756.744623] LNet: Removed LNI 10.1.2.3@tcp [ 757.251501] Key type .llcrypt unregistered [ 757.255095] Key type ._llcrypt unregistered [ 762.475183] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 762.756738] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 762.761249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 762.765774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 764.412597] Key type ._llcrypt registered [ 764.416311] Key type .llcrypt registered [ 764.607329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.618181] alg: No test for adler32 (adler32-zlib) [ 765.529271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.580616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 772.919520] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 08:52:52 (1774875172) [ 773.465950] LNet: 55091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.472639] LNetError: 55091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 773.494173] LNet: Removed LNI 10.1.2.3@tcp [ 773.959909] Key type .llcrypt unregistered [ 773.964501] Key type ._llcrypt unregistered [ 778.947548] Key type ._llcrypt registered [ 778.950966] Key type .llcrypt registered [ 779.119183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 779.135227] alg: No test for adler32 (adler32-zlib) [ 780.017206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 780.042514] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 786.042500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 786.614113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_interrupt -r 1 [ 786.664142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_interrupt -r 1 [ 786.698337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 786.717459] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 786.908400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 786.949405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 786.981242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 787.009182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 787.049665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 787.078324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 787.104371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 787.129151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 787.146950] LNet: 55529:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 787.163982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 787.198613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 787.233557] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 787.237966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 787.239060] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 787.275691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 787.297433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 787.327630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 787.374580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 787.769243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_dropped -r 1 [ 787.797162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_dropped -r 1 [ 787.829473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 788.018201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 788.038767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 788.070546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 788.096328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 788.119119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 788.140807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 788.163180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 788.185987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 788.200566] LNet: 55638:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 788.214392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 788.239238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 788.267558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 788.290514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 788.318408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 788.339853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 788.365747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 788.791468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_aborted -r 1 [ 788.836749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_aborted -r 1 [ 788.883668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 788.907436] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 788.911542] LNet: Skipped 1 previous similar message [ 789.074874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 789.099100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 789.125272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 789.155796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 789.192193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 789.227074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 789.252507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 789.275916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 789.301133] LNet: 55747:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 789.323455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 789.348287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 789.373282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 789.398334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 789.428861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 789.462403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 789.487921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 789.823039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_no_route -r 1 [ 789.852242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_no_route -r 1 [ 789.871969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 790.024679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 790.050719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 790.079916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 790.103809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 790.127043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 790.165332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 790.198921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 790.240316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 790.270085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 790.305397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 790.329625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 790.359919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 790.388436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 790.418412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 790.457957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 790.828647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_timeout -r 1 [ 790.863382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_timeout -r 1 [ 790.903495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 790.933963] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 790.939175] LNet: Skipped 2 previous similar messages [ 791.088411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 791.114421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 791.146554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 791.171506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 791.204769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 791.230383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 791.258382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 791.294190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 791.307380] LNet: 55965:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 791.312429] LNet: 55965:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 791.326328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 791.360289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 791.382357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 791.410078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 791.441143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 791.464534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 791.502936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 791.910395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_error -r 1 [ 791.941495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_error -r 1 [ 791.967863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 792.137256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 792.167647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 792.214471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 792.249390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 792.278639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 792.312781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 792.358806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 792.391790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 792.415953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 792.443900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 792.484376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 792.523829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 792.546848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 792.571924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 792.598626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 792.784931] LNet: 56116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.788939] LNetError: 56116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.807687] LNet: Removed LNI 192.168.206.9@tcp [ 793.147667] Key type .llcrypt unregistered [ 793.150829] Key type ._llcrypt unregistered [ 805.576061] Key type ._llcrypt registered [ 805.577450] Key type .llcrypt registered [ 806.728924] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 08:53:26 (1774875206) [ 807.564419] Key type .llcrypt unregistered [ 807.570105] Key type ._llcrypt unregistered [ 812.454628] Key type ._llcrypt registered [ 812.456779] Key type .llcrypt registered [ 812.699353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 812.733773] alg: No test for adler32 (adler32-zlib) [ 813.658930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 813.705756] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 820.823758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 830.302872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.352559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 830.383524] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 832.038815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_interrupt -r 1 [ 832.088240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_interrupt -r 1 [ 832.130913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_interrupt -r 1 [ 832.175940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_interrupt -r 1 [ 832.225386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_interrupt -r 1 [ 832.266413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_interrupt -r 1 [ 832.308884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_interrupt -r 1 [ 832.363375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_interrupt -r 1 [ 832.402515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 832.430372] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 832.440382] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 832.646281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 832.672572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 832.704797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 832.746866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 832.787259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 832.818979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 832.857150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 832.893596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 832.922818] LNet: 57580:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 832.945075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 832.984644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 833.026780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 833.067336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 833.111529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 833.147768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 833.194646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 833.732187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_dropped -r 1 [ 833.773415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_dropped -r 1 [ 833.802921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_dropped -r 1 [ 833.850663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_dropped -r 1 [ 833.894823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_dropped -r 1 [ 833.940653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_dropped -r 1 [ 833.969056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_dropped -r 1 [ 834.015047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_dropped -r 1 [ 834.060242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 834.085683] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 834.094126] LNet: Skipped 2 previous similar messages [ 834.277365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 834.300247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 834.322283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 834.353664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 834.390858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 834.416579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 834.445801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 834.477836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 834.489174] LNet: 57702:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 834.503585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 834.537131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 834.576850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 834.603913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 834.633039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 834.664919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 834.685992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 835.137671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_aborted -r 1 [ 835.175939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_aborted -r 1 [ 835.210801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_aborted -r 1 [ 835.256358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_aborted -r 1 [ 835.306605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_aborted -r 1 [ 835.343238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_aborted -r 1 [ 835.385559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_aborted -r 1 [ 835.414973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_aborted -r 1 [ 835.441115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 835.459398] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 835.464253] LNet: Skipped 2 previous similar messages [ 835.641169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 835.667084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 835.690419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 835.713678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 835.746479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 835.775283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 835.796907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 835.831495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 835.852811] LNet: 57823:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 835.869099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 835.892398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 835.919293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 835.940306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 835.963454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 835.995493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 836.034115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 836.562554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_no_route -r 1 [ 836.629065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_no_route -r 1 [ 836.683410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_no_route -r 1 [ 836.736989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_no_route -r 1 [ 836.786616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_no_route -r 1 [ 836.849134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_no_route -r 1 [ 836.906832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_no_route -r 1 [ 836.958966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_no_route -r 1 [ 837.016874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 837.235435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 837.260316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 837.286729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 837.317735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 837.349652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 837.382507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 837.411829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 837.454270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 837.494786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 837.525328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 837.563167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 837.601825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 837.628403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 837.658861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 837.687531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 838.165937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_timeout -r 1 [ 838.201397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_timeout -r 1 [ 838.235639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_timeout -r 1 [ 838.264321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_timeout -r 1 [ 838.290882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_timeout -r 1 [ 838.323159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_timeout -r 1 [ 838.355329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_timeout -r 1 [ 838.386209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_timeout -r 1 [ 838.417434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 838.440901] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 838.446197] LNet: Skipped 5 previous similar messages [ 838.632768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 838.654382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 838.693629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 838.727865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 838.771281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 838.802676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 838.840361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 838.887316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 838.914906] LNet: 58065:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 838.925810] LNet: 58065:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 838.948754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 838.986386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 839.010135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 839.046995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 839.081488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.123478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.173970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 839.640733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_error -r 1 [ 839.681311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_error -r 1 [ 839.727714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_error -r 1 [ 839.775859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_error -r 1 [ 839.827151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_error -r 1 [ 839.866519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_error -r 1 [ 839.897692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_error -r 1 [ 839.930811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_error -r 1 [ 839.972440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 840.191703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 840.222435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 840.261583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 840.292283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 840.319489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 840.351250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 840.384948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 840.440587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 840.465422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 840.505928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 840.551640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 840.572289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 840.603160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 840.632513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 840.659309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 845.341140] LNet: 58262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.346611] LNetError: 58262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.351138] LNet: Removed LNI 192.168.206.9@tcp [ 846.383829] LNet: Removed LNI 192.168.206.9@tcp1 [ 846.791677] Key type .llcrypt unregistered [ 846.794614] Key type ._llcrypt unregistered [ 860.469135] Key type ._llcrypt registered [ 860.471275] Key type .llcrypt registered [ 861.462352] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 08:54:21 (1774875261) [ 862.318335] Key type .llcrypt unregistered [ 862.322772] Key type ._llcrypt unregistered [ 866.912414] Key type ._llcrypt registered [ 866.916958] Key type .llcrypt registered [ 867.098330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 867.128124] alg: No test for adler32 (adler32-zlib) [ 868.036609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 868.065404] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 873.793311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 874.280047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_dropped -r 1 [ 874.312869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_dropped -r 1 [ 874.347094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 874.365748] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 874.534122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 874.565106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 874.594085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 874.622682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 874.643531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 874.667169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 874.698651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 874.724844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 874.738959] LNet: 59509:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 874.753680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 874.786935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 874.829620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 874.860095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 874.884686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 874.910517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 874.937162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 875.317453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_error -r 1 [ 875.343770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_error -r 1 [ 875.370826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 875.393601] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 875.548073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 875.565885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 875.582723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 875.603203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 875.632751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 875.654382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 875.690760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 875.718408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 875.737188] LNet: 59619:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 875.751485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 875.771878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 875.801456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 875.837470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 875.867935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 875.897840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 875.916631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 876.267342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_timeout -r 1 [ 876.297435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_timeout -r 1 [ 876.336254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 876.503954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 876.542136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 876.575738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 876.611953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 876.653939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 876.680700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 876.706422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 876.730831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 876.745512] LNet: 59728:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 876.763387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 876.791625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 876.830557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 876.864399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 876.897338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 876.929966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 876.951432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 877.162230] LNet: 59769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.169590] LNetError: 59769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.183840] LNet: Removed LNI 192.168.206.9@tcp [ 877.460043] Key type .llcrypt unregistered [ 877.462388] Key type ._llcrypt unregistered [ 887.739940] Key type ._llcrypt registered [ 887.742415] Key type .llcrypt registered [ 889.061863] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 08:54:48 (1774875288) [ 889.707583] Key type .llcrypt unregistered [ 889.710663] Key type ._llcrypt unregistered [ 893.573304] Key type ._llcrypt registered [ 893.575097] Key type .llcrypt registered [ 893.701792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.713171] alg: No test for adler32 (adler32-zlib) [ 894.588625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 894.607288] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 899.924512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 906.057301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.092151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 906.109595] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 907.213709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_dropped -r 1 [ 907.242395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e remote_dropped -r 1 [ 907.276140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_dropped -r 1 [ 907.317202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e remote_dropped -r 1 [ 907.346144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e remote_dropped -r 1 [ 907.382036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e remote_dropped -r 1 [ 907.410559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e remote_dropped -r 1 [ 907.446019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e remote_dropped -r 1 [ 907.491519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 907.517534] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 907.528163] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.109@tcp [ 907.723696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 907.745650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 907.774789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 907.799123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 907.826261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 907.850549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 907.873860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 907.899692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 907.918615] LNet: 61245:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 907.930163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 907.961767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 907.981937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 908.017112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 908.052905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 908.079980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 908.104500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 908.131548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 908.482907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_error -r 1 [ 908.520750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e remote_error -r 1 [ 908.558799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_error -r 1 [ 908.590217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e remote_error -r 1 [ 908.620114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e remote_error -r 1 [ 908.649843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e remote_error -r 1 [ 908.682482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e remote_error -r 1 [ 908.709141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e remote_error -r 1 [ 908.738718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 908.756382] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 908.760297] LNet: Skipped 2 previous similar messages [ 908.942651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 908.979271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 909.022296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 909.051652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 909.101689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 909.129240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 909.162363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 909.196540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 909.219425] LNet: 61368:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 909.243378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 909.270550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 909.306485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 909.334863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 909.387620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 909.423583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 909.445486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 909.482153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 909.925423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_timeout -r 1 [ 909.970325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e remote_timeout -r 1 [ 909.997343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_timeout -r 1 [ 910.023478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e remote_timeout -r 1 [ 910.071038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e remote_timeout -r 1 [ 910.113333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e remote_timeout -r 1 [ 910.162386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e remote_timeout -r 1 [ 910.201066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e remote_timeout -r 1 [ 910.245637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.109@tcp [ 910.268232] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 910.428138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 910.463277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 910.515517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 910.560763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 910.594549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 910.645449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 910.689489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 910.715983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 910.737707] LNet: 61492:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 910.761937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 910.788407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 910.827515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 910.870338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 910.893494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 910.928150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 910.960769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 910.996040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 915.503776] LNet: 61571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.509784] LNetError: 61571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.516179] LNet: Removed LNI 192.168.206.9@tcp [ 915.524745] LNet: Removed LNI 192.168.206.9@tcp1 [ 915.794385] Key type .llcrypt unregistered [ 915.801347] Key type ._llcrypt unregistered [ 928.316901] Key type ._llcrypt registered [ 928.321203] Key type .llcrypt registered [ 929.650904] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 08:55:29 (1774875329) [ 930.078544] Key type .llcrypt unregistered [ 930.082687] Key type ._llcrypt unregistered [ 934.268797] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 934.382543] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 934.398346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 934.406745] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 936.883389] Key type ._llcrypt registered [ 936.886235] Key type .llcrypt registered [ 937.012061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.030335] alg: No test for adler32 (adler32-zlib) [ 937.940664] LNet: 62647:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 937.952383] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 937.989974] LNet: 62661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.998380] LNetError: 62661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.012315] LNet: Removed LNI 192.168.206.9@tcp [ 938.488390] Key type .llcrypt unregistered [ 938.489919] Key type ._llcrypt unregistered [ 942.713535] Key type ._llcrypt registered [ 942.715480] Key type .llcrypt registered [ 942.885074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 942.898438] alg: No test for adler32 (adler32-zlib) [ 943.800344] LNet: 62920:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 943.823100] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 943.826229] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 943.850781] LNet: 62934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.859762] LNetError: 62934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.869090] LNet: Removed LNI 192.168.206.9@tcp [ 943.885280] LNet: Removed LNI 10.1.2.3@tcp1 [ 944.399529] Key type .llcrypt unregistered [ 944.401880] Key type ._llcrypt unregistered [ 948.559609] Key type ._llcrypt registered [ 948.563518] Key type .llcrypt registered [ 948.688768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 948.707307] alg: No test for adler32 (adler32-zlib) [ 949.605473] LNet: 63207:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 949.618391] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 949.621681] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 949.653750] LNet: 63221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.661977] LNetError: 63221:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.668515] LNet: Removed LNI 192.168.206.9@tcp [ 949.685418] LNet: Removed LNI 10.1.2.3@tcp [ 950.254570] Key type .llcrypt unregistered [ 950.256522] Key type ._llcrypt unregistered [ 954.527457] Key type ._llcrypt registered [ 954.529678] Key type .llcrypt registered [ 954.712513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 954.727432] alg: No test for adler32 (adler32-zlib) [ 955.614099] LNet: 63480:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 955.623195] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 955.626464] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 955.652966] LNet: 63494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.657820] LNetError: 63494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.663601] LNet: Removed LNI 192.168.206.9@tcp [ 955.679901] LNet: Removed LNI 10.1.2.3@tcp [ 956.181646] Key type .llcrypt unregistered [ 956.183802] Key type ._llcrypt unregistered [ 960.702669] Key type ._llcrypt registered [ 960.704583] Key type .llcrypt registered [ 960.857812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.876779] alg: No test for adler32 (adler32-zlib) [ 961.779943] LNet: 63758:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 961.801211] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 961.805544] LNetError: 63758:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 961.810833] LNetError: Error -100 starting up LNI tcp [ 961.820399] LNet: Removed LNI 192.168.206.9@tcp [ 962.306523] Key type .llcrypt unregistered [ 962.314086] Key type ._llcrypt unregistered [ 971.511561] Key type ._llcrypt registered [ 971.513556] Key type .llcrypt registered [ 972.677448] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 08:56:12 (1774875372) [ 973.239295] Key type .llcrypt unregistered [ 973.242058] Key type ._llcrypt unregistered [ 977.796953] Key type ._llcrypt registered [ 977.800697] Key type .llcrypt registered [ 977.979301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 977.996146] alg: No test for adler32 (adler32-zlib) [ 978.876645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 978.904752] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 984.328781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 984.886670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e network_timeout -r 1 [ 984.920168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e network_timeout -r 1 [ 984.949406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 984.962143] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 985.055818] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 985.158337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 985.194934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 985.221206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 985.247098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 985.271306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 985.297441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 985.335585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 985.366939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 985.387329] LNet: 65013:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 985.414845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 985.447469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 985.480869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 985.526302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 985.570904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 985.605786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 985.802488] LNet: 65052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.806728] LNetError: 65052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.819276] LNet: Removed LNI 192.168.206.9@tcp [ 986.118792] Key type .llcrypt unregistered [ 986.121574] Key type ._llcrypt unregistered [ 1000.361823] Key type ._llcrypt registered [ 1000.364214] Key type .llcrypt registered [ 1000.549889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1000.570340] alg: No test for adler32 (adler32-zlib) [ 1001.453484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1001.480593] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1007.452673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1014.093025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1014.131301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1014.156260] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1015.266212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e network_timeout -r 1 [ 1015.304600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e network_timeout -r 1 [ 1015.332475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e network_timeout -r 1 [ 1015.354956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e network_timeout -r 1 [ 1015.392725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e network_timeout -r 1 [ 1015.415640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e network_timeout -r 1 [ 1015.444412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e network_timeout -r 1 [ 1015.474292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e network_timeout -r 1 [ 1015.497364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1015.510271] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1015.651317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.684677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1015.720399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1015.753806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1015.786653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1015.833439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1015.839178] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1015.851377] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.109@tcp [ 1015.875397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1015.917640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1015.935323] LNet: 65946:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1015.955965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1015.990618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.024642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.062296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.084304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.109496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.934752] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1020.196766] LNet: 66019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.204614] LNetError: 66019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.212516] LNet: Removed LNI 192.168.206.9@tcp [ 1020.231886] LNet: Removed LNI 192.168.206.9@tcp1 [ 1020.532772] Key type .llcrypt unregistered [ 1020.535295] Key type ._llcrypt unregistered [ 1034.226438] Key type ._llcrypt registered [ 1034.228307] Key type .llcrypt registered [ 1035.399633] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 08:57:15 (1774875435) [ 1035.795975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1035.818634] alg: No test for adler32 (adler32-zlib) [ 1036.796733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.860633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1036.894557] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1037.756209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1037.770426] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1038.557694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1038.638977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1038.675388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1038.706821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1038.740994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1038.773377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1038.807944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.9@tcp [ 1038.837943] LNetError: 67120:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.9@tcp to recovery [ 1038.879351] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1038.879421] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1040.991560] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1041.211496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1044.063540] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1044.734524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1044.781478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1044.888690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1044.917681] LNet: 67217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.926511] LNetError: 67217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.938267] LNet: Removed LNI 192.168.206.9@tcp [ 1044.950731] LNet: Removed LNI 192.168.206.9@tcp1 [ 1044.980379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.067765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1045.097524] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1045.751464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1045.765263] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1046.535382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1046.569319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1046.610351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1046.649236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1046.678935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1046.721766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1046.759921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1046.792467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.9@tcp [ 1046.816304] LNetError: 67489:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.9@tcp to recovery [ 1047.077833] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1050.171862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1053.215447] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1053.219477] LNet: Skipped 1 previous similar message [ 1057.878602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1057.912879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1057.944825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.887905] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 08:57:43 (1774875463) [ 1064.008295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1064.042558] LNet: 68203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.047856] LNetError: 68203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.053610] LNetError: 68203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1064.061742] LNet: Removed LNI 192.168.206.9@tcp [ 1064.086491] LNet: Removed LNI 192.168.206.9@tcp1 [ 1064.119736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.169960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1064.195403] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1064.805456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1064.833089] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1065.657704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1065.766506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1065.808798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1065.853429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1065.912346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1065.943792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1065.981514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.9@tcp --health 0 [ 1066.096650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1066.207099] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.9@tcp [ 1067.231365] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1067.239510] LNet: Skipped 1 previous similar message [ 1082.482215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1082.634436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1082.675495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.720627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.9@tcp --health 500 [ 1082.835358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1089.559212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1089.621845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1089.639389] LNet: 68707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.647889] LNetError: 68707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.655908] LNetError: 68707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1089.661743] LNet: Removed LNI 192.168.206.9@tcp [ 1089.670154] LNet: Removed LNI 192.168.206.9@tcp1 [ 1089.722967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.776376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1089.796950] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1090.561845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1090.580259] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1091.289776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1091.320211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.391200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1091.422996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1091.460624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1091.501355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1091.544839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1091.594899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.9@tcp --health 0 [ 1092.832307] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1092.836887] LNet: Skipped 3 previous similar messages [ 1094.034788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1103.018511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1103.057922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.109804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.278420] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 08:58:29 (1774875509) [ 1109.821649] LNet: 69752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.830417] LNetError: 69752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.840885] LNetError: 69752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1109.854080] LNet: Removed LNI 192.168.206.9@tcp [ 1109.863553] LNet: Removed LNI 192.168.206.9@tcp1 [ 1110.561840] Key type .llcrypt unregistered [ 1110.563987] Key type ._llcrypt unregistered [ 1115.892474] Key type ._llcrypt registered [ 1115.895266] Key type .llcrypt registered [ 1116.071460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1116.085170] alg: No test for adler32 (adler32-zlib) [ 1117.025469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1117.054696] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1125.018549] Lustre: DEBUG MARKER: Initial discovery [ 1125.098745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.109@tcp [ 1127.078315] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1127.112038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1127.145887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.109@tcp [ 1127.172485] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1127.201189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1127.241591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.109@tcp -r 1 -m GET -l 3 [ 1127.275528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.109@tcp -r 1 -m GET -e local_error [ 1127.308879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.109@tcp -r 1 -m PUT -l 6 [ 1127.327376] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1128.546219] Lustre: DEBUG MARKER: Force oleg609-client.virtnet to discover 192.168.206.109@tcp (in background) [ 1128.603378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.109@tcp [ 1129.902201] Lustre: DEBUG MARKER: Force oleg609-server to discover 192.168.206.9@tcp [ 1131.681437] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1132.138933] Lustre: DEBUG MARKER: Wait for 70506 [ 1133.520985] Lustre: DEBUG MARKER: Finished wait on 70506 [ 1133.600836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1133.642167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1133.648659] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1133.937803] LNet: 70963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.952068] LNetError: 70963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.975273] LNet: Removed LNI 192.168.206.9@tcp [ 1134.315742] Key type .llcrypt unregistered [ 1134.321782] Key type ._llcrypt unregistered [ 1149.382136] Key type ._llcrypt registered [ 1149.385579] Key type .llcrypt registered [ 1150.705307] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 08:59:10 (1774875550) [ 1151.413536] Key type .llcrypt unregistered [ 1151.417929] Key type ._llcrypt unregistered [ 1157.675905] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1157.739349] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1157.744868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1157.749092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1160.448381] Key type ._llcrypt registered [ 1160.451520] Key type .llcrypt registered [ 1160.644689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.660882] alg: No test for adler32 (adler32-zlib) [ 1161.550609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1161.595850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1161.613883] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1162.241861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1162.268891] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1170.073200] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 08:59:29 (1774875569) [ 1170.211837] LNet: 73031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.218502] LNetError: 73031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.224798] LNet: Removed LNI 192.168.206.9@tcp [ 1170.264020] LNet: Removed LNI 10.1.2.3@tcp [ 1170.818605] Key type .llcrypt unregistered [ 1170.822271] Key type ._llcrypt unregistered [ 1176.222446] Key type ._llcrypt registered [ 1176.227602] Key type .llcrypt registered [ 1176.374694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1176.395819] alg: No test for adler32 (adler32-zlib) [ 1177.369925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1177.437427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1177.453837] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1178.230352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1178.256169] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1178.947526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.9@tcp [ 1179.999235] LNetError: Unexpected error -22 connecting to 192.168.206.9@tcp at host 192.168.206.9:988 [ 1184.105868] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 08:59:43 (1774875583) [ 1184.187477] LNet: 74158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.192251] LNetError: 74158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.199772] LNet: Removed LNI 192.168.206.9@tcp [ 1184.209330] LNet: Removed LNI 10.1.2.3@tcp [ 1184.943830] Key type .llcrypt unregistered [ 1184.945461] Key type ._llcrypt unregistered [ 1189.265084] Key type ._llcrypt registered [ 1189.266638] Key type .llcrypt registered [ 1189.388295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1189.401473] alg: No test for adler32 (adler32-zlib) [ 1190.265723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1190.315477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1190.332986] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1190.888686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1190.907988] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1191.610906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.9@tcp1 --nid 192.168.206.9@tcp2 [ 1194.974402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp1 192.168.206.9@tcp2 [ 1194.995825] LNetError: 75001:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.9@tcp1 to 192.168.206.9@tcp2 - Route table may be misconfigured [ 1195.026892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp2 192.168.206.9@tcp1 [ 1199.542565] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 08:59:59 (1774875599) [ 1199.644409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1199.663584] LNet: 75572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.673912] LNetError: 75572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.684235] LNet: Removed LNI 192.168.206.9@tcp1 [ 1200.744339] LNet: Removed LNI 192.168.206.9@tcp2 [ 1200.764975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1200.825205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1200.859439] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1201.428517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1201.443394] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1202.001777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1202.035933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.9@tcp [ 1202.062447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp1 192.168.206.9@tcp1 [ 1202.093866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.9@tcp -d 192.168.206.9@tcp -e network_timeout [ 1202.132792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -e network_timeout [ 1202.172461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -e network_timeout [ 1202.209901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -e network_timeout [ 1202.241859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1202.274227] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1202.292125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1202.374492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1202.445279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1202.847651] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1207.679556] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:00:07 (1774875607) [ 1207.821299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1207.848063] LNet: 76433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.859867] LNetError: 76433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.867457] LNetError: 76433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1207.880048] LNet: Removed LNI 192.168.206.9@tcp [ 1208.934327] LNet: Removed LNI 192.168.206.9@tcp1 [ 1208.973331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.073460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1209.312880] LNet: 76472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1209.711784] Key type .llcrypt unregistered [ 1209.716762] Key type ._llcrypt unregistered [ 1220.236846] Key type ._llcrypt registered [ 1220.239087] Key type .llcrypt registered [ 1221.614512] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:00:21 (1774875621) [ 1222.111216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1222.142962] alg: No test for adler32 (adler32-zlib) [ 1223.169070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1224.532394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1226.015708] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:00:25 (1774875625) [ 1226.104911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1226.129231] LNet: 77567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.153873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1226.292870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1226.325420] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1226.976459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1227.008839] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1227.832416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1227.859985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp1 [ 1227.889938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp1 [ 1233.135104] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:00:33 (1774875633) [ 1234.621870] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg609-server [ 1236.138746] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:00:35 (1774875635) [ 1237.827651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg609-server [ 1239.375316] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:00:39 (1774875639) [ 1241.189174] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg609-server [ 1242.933657] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:00:42 (1774875642) [ 1244.693167] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg609-server [ 1246.209424] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:00:46 (1774875646) [ 1247.756920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg609-server [ 1249.916662] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:00:49 (1774875649) [ 1252.450291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg609-server [ 1254.400959] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:00:53 (1774875653) [ 1255.887048] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg609-server [ 1257.423730] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:00:57 (1774875657) [ 1259.141322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg609-server [ 1260.806875] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:01:00 (1774875660) [ 1262.545598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg609-server [ 1263.821899] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:01:03 (1774875663) [ 1264.041052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1264.063482] LNet: 81458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.070730] LNetError: 81458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.076288] LNet: Removed LNI 192.168.206.9@tcp [ 1265.125222] LNet: Removed LNI 192.168.206.9@tcp1 [ 1265.145921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1265.200593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1265.217461] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1265.779102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1265.861757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1266.019955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1266.043895] LNet: 81606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1266.047758] LNetError: 81606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.056797] LNetError: 81606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1266.073683] LNet: Removed LNI 192.168.206.9@tcp [ 1266.096932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1266.158406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1266.189547] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1266.927655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1267.090765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1267.121119] LNetError: 81751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1267.146682] LNet: Removed LNI 192.168.206.9@tcp [ 1267.170984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1267.234961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1267.254243] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1267.968319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1268.073629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1268.310183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1268.331699] LNet: 81898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.337255] LNet: 81898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1269.414430] LNet: Removed LNI 192.168.206.9@tcp [ 1269.451768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.534313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1269.567413] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1270.331812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1270.555950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1270.590316] LNetError: 82044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.596493] LNetError: 82044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1271.664501] LNet: Removed LNI 192.168.206.9@tcp [ 1271.689255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1271.799721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1271.840240] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1272.735675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1272.829541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1273.076947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1273.108862] LNet: 82191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.116130] LNet: 82191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1274.150406] LNet: Removed LNI 192.168.206.9@tcp [ 1274.173645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1274.236655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1274.279466] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1275.201830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1275.560582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1275.591516] LNetError: 82336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.597532] LNetError: 82336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1275.614845] LNet: Removed LNI 192.168.206.9@tcp [ 1275.634888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.706709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1275.762134] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1276.656476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1276.712206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1277.007964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1278.119075] LNet: Removed LNI 192.168.206.9@tcp [ 1278.170846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.247226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1278.308264] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1279.085886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1279.375625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1280.424546] LNet: Removed LNI 192.168.206.9@tcp [ 1280.455435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.525162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1280.558444] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1281.149473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1281.215565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1281.430652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1281.453164] LNet: 82776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1281.457815] LNet: 82776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1282.539537] LNet: Removed LNI 192.168.206.9@tcp [ 1282.578796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1282.635246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1282.668486] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1283.317143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1283.526630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1284.589465] LNet: Removed LNI 192.168.206.9@tcp [ 1284.622304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1284.697433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1284.729496] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1285.390729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1285.463064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1288.028899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1288.048764] LNetError: 83087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.054432] LNetError: 83087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1289.132779] LNet: Removed LNI 192.168.206.9@tcp [ 1289.164381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1289.218637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1289.247402] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1289.981302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1290.292441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1291.366803] LNet: Removed LNI 192.168.206.9@tcp [ 1291.386637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1291.425878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1291.453410] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1292.041173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1292.110681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1292.144585] LNet: Removed LNI 192.168.206.9@tcp [ 1292.157470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.197435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1292.235592] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1292.886171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1299.157058] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:01:38 (1774875698) [ 1299.255320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1299.303561] LNet: 84061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.315869] LNet: 84061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1299.349689] LNet: Removed LNI 192.168.206.9@tcp [ 1299.374520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1299.439585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1299.471456] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/180] [ 1300.495572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1300.524256] LNet: Removed LNI 192.168.206.9@tcp231 [ 1300.558624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.633107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1300.685113] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/180] [ 1300.700437] LNet: 84215:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1301.013575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1301.055262] LNet: Removed LNI 192.168.206.9@tcp231 [ 1301.072256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1301.150337] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/180] [ 1302.055326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1302.136905] LNet: Removed LNI 192.168.206.9@tcp231 [ 1302.175854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.357918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1302.397620] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/0] [ 1302.400645] LNet: 84367:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.666777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1302.690776] LNet: Removed LNI 192.168.206.9@tcp231 [ 1302.709602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1302.766915] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/0] [ 1304.231384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1304.250942] LNetError: 84510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.257315] LNetError: 84510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1304.290843] LNet: Removed LNI 192.168.206.9@tcp231 [ 1304.325424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.390617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1304.423745] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/60] [ 1304.429964] LNet: 84518:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1304.858220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1304.899517] LNet: Removed LNI 192.168.206.9@tcp231 [ 1304.933588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1304.989988] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/60] [ 1306.200081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1306.239760] LNet: Removed LNI 192.168.206.9@tcp231 [ 1306.277371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1306.402934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1306.484110] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/180] [ 1306.491365] LNet: 84670:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1307.347380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1307.397467] LNet: Removed LNI 192.168.206.9@tcp231 [ 1307.419706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1307.471662] LNet: Added LNI 192.168.206.9@tcp231 [8/256/0/180] [ 1313.986895] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:01:53 (1774875713) [ 1314.171111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1314.229115] LNet: Removed LNI 192.168.206.9@tcp231 [ 1314.274963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1314.320074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1314.337224] LNet: Added LNI 192.168.206.9@tcp232 [8/256/0/180] [ 1314.888414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1314.945272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1314.993316] LNet: Removed LNI 192.168.206.9@tcp232 [ 1315.014180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1315.038314] LNet: Added LNI 192.168.206.9@tcp232 [8/256/0/180] [ 1315.718337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1315.756286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1315.792939] LNet: Removed LNI 192.168.206.9@tcp232 [ 1315.817340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1315.844614] LNet: Added LNI 192.168.206.9@tcp232 [8/256/0/180] [ 1316.452812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1316.491593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1316.526821] LNet: Removed LNI 192.168.206.9@tcp232 [ 1316.547321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1316.589324] LNet: Added LNI 192.168.206.9@tcp232 [8/256/0/180] [ 1317.229524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1317.261874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1317.310248] LNet: Removed LNI 192.168.206.9@tcp232 [ 1317.336196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1322.796896] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:02:02 (1774875722) [ 1324.048694] Key type .llcrypt unregistered [ 1324.053135] Key type ._llcrypt unregistered [ 1329.383170] Key type ._llcrypt registered [ 1329.385175] Key type .llcrypt registered [ 1329.591395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1329.604141] alg: No test for adler32 (adler32-zlib) [ 1330.497643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1330.519641] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1336.654830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1343.889714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1343.921471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1343.947618] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1344.952669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_interrupt -r 2 [ 1344.981589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_interrupt -r 2 [ 1345.016798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_interrupt -r 2 [ 1345.053600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_interrupt -r 2 [ 1345.105964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_interrupt -r 2 [ 1345.149933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_interrupt -r 2 [ 1345.201436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_interrupt -r 2 [ 1345.248576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_interrupt -r 2 [ 1345.296174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1345.313593] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1345.318901] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1345.526645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1345.565615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1345.592958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1345.620680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1345.674437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1345.712690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1345.752570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1345.808670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1345.842672] LNet: 87186:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1345.873144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1345.899654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1345.924771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1345.951404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1345.994509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1346.031063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1346.546984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_dropped -r 2 [ 1346.577728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_dropped -r 2 [ 1346.607635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_dropped -r 2 [ 1346.650151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_dropped -r 2 [ 1346.678531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_dropped -r 2 [ 1346.734887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_dropped -r 2 [ 1346.769665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_dropped -r 2 [ 1346.820589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_dropped -r 2 [ 1346.890693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1346.930932] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1347.200798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1347.231908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1347.296397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1347.355556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1347.396577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1347.421727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1347.440858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1347.473661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1347.486908] LNet: 87307:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1347.509214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1347.554584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1347.579487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1347.607388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1347.645806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1347.688769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1348.287798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_aborted -r 2 [ 1348.358915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_aborted -r 2 [ 1348.407313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_aborted -r 2 [ 1348.443967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_aborted -r 2 [ 1348.472045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_aborted -r 2 [ 1348.497208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_aborted -r 2 [ 1348.536118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_aborted -r 2 [ 1348.587357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_aborted -r 2 [ 1348.625256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1348.657180] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1348.959166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.987847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1349.013739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1349.039334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1349.084231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1349.121147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1349.163538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1349.195346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1349.212897] LNet: 87427:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1349.234741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1349.258881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1349.287794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.318448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.355755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.389103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.950490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_no_route -r 2 [ 1349.981653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_no_route -r 2 [ 1350.021340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_no_route -r 2 [ 1350.063692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_no_route -r 2 [ 1350.098731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_no_route -r 2 [ 1350.148729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_no_route -r 2 [ 1350.197311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_no_route -r 2 [ 1350.248212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_no_route -r 2 [ 1350.293309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1350.597341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.636439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1350.670865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.706989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.737739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1350.781254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1350.812478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1350.838646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1350.878826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1350.929658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1351.006349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1351.033329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1351.055956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.086850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1351.496038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e local_timeout -r 2 [ 1351.586883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e local_timeout -r 2 [ 1351.643563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e local_timeout -r 2 [ 1351.672158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e local_timeout -r 2 [ 1351.715569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e local_timeout -r 2 [ 1351.763836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e local_timeout -r 2 [ 1351.802758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e local_timeout -r 2 [ 1351.892432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e local_timeout -r 2 [ 1351.977108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1352.011625] LNet: There was an unexpected network error while writing to 192.168.206.109: rc = -22 [ 1352.019134] LNet: Skipped 1 previous similar message [ 1352.369934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1352.439307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1352.498559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1352.528104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1352.555757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1352.595791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1352.632694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1352.682534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1352.729799] LNet: 87668:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1352.751733] LNet: 87668:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1352.796307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1352.862835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1352.965454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1353.062550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1353.106478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1353.179237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1353.873080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp -m GET -e remote_dropped -r 2 [ 1353.922908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.109@tcp1 -m GET -e remote_dropped -r 2 [ 1353.987607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp -m GET -e remote_dropped -r 2 [ 1354.105786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp -d 192.168.206.9@tcp1 -m GET -e remote_dropped -r 2 [ 1354.150651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp -m GET -e remote_dropped -r 2 [ 1354.210406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.109@tcp1 -m GET -e remote_dropped -r 2 [ 1354.265735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp -m GET -e remote_dropped -r 2 [ 1354.300122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.9@tcp1 -d 192.168.206.9@tcp1 -m GET -e remote_dropped -r 2 [ 1354.339682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.9@tcp 192.168.206.109@tcp [ 1354.362510] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.109@tcp [ 1354.676660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1354.756597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1354.800874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1354.874146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1354.933746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1354.987641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1355.032447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1355.073778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1355.119394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1355.157321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1355.193744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1355.235465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1355.272930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1355.334342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1361.121615] LNet: 87862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.140025] LNetError: 87862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1361.149199] LNet: Removed LNI 192.168.206.9@tcp [ 1361.167467] LNet: Removed LNI 192.168.206.9@tcp1 [ 1361.415639] Key type .llcrypt unregistered [ 1361.418795] Key type ._llcrypt unregistered [ 1378.814190] Key type ._llcrypt registered [ 1378.816526] Key type .llcrypt registered [ 1380.606687] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:02:59 (1774875779) [ 1381.265573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1381.298353] alg: No test for adler32 (adler32-zlib) [ 1382.411932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.521123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1382.565104] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1383.679549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1383.705799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.9@tcp [ 1383.725396] LNetError: 88843:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.9@tcp to recovery [ 1383.871242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.9@tcp [ 1384.034161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1393.311790] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:03:11 (1774875791) [ 1394.255909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1394.312419] LNet: 89434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.335250] LNetError: 89434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.351992] LNet: Removed LNI 192.168.206.9@tcp [ 1394.390944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.466553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1394.494743] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1395.227988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1395.247754] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1396.271218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.9@tcp [ 1396.307793] LNetError: 89700:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.9@tcp to recovery [ 1396.511502] LNet: 1 local NIs in recovery (showing 1): 192.168.206.9@tcp [ 1396.513047] LNet: There was an unexpected network error while writing to 192.168.206.9: rc = -22 [ 1397.657617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.9@tcp [ 1406.053069] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:03:25 (1774875805) [ 1407.821633] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg609-server [ 1409.985238] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:03:29 (1774875809) [ 1411.544986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg609-server [ 1413.013502] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:03:32 (1774875812) [ 1414.598297] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg609-server [ 1416.288486] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:03:36 (1774875816) [ 1416.378268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1416.398765] LNet: 91318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1416.410766] LNetError: 91318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1416.420862] LNet: Removed LNI 192.168.206.9@tcp [ 1416.431495] LNet: Removed LNI 192.168.206.9@tcp1 [ 1416.504068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.587152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1416.623293] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1417.299540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1417.381941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1417.444457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1417.555898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1417.612814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1417.682615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1423.447440] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:03:43 (1774875823) [ 1423.552970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1423.576638] LNet: 92041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.581502] LNetError: 92041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.589087] LNetError: 92041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1423.600984] LNet: Removed LNI 192.168.206.9@tcp [ 1423.621364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.754418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1423.786316] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1429.842295] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:03:49 (1774875829) [ 1429.910654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1429.945419] LNet: 92744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.951506] LNetError: 92744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1429.972802] LNet: Removed LNI 192.168.206.9@tcp [ 1430.004996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.083689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1430.101595] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1435.458578] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:03:55 (1774875835) [ 1436.436730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1437.643806] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:03:57 (1774875837) [ 1438.015893] LNet: 93786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.020741] LNetError: 93786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1438.036731] LNet: Removed LNI 192.168.206.9@tcp [ 1438.525893] Key type .llcrypt unregistered [ 1438.528282] Key type ._llcrypt unregistered [ 1443.065159] Key type ._llcrypt registered [ 1443.069267] Key type .llcrypt registered [ 1443.215398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1443.229855] alg: No test for adler32 (adler32-zlib) [ 1444.135543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1444.155972] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1450.196587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1454.293909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.109@tcp [ 1454.538141] LNet: 94203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.544245] LNetError: 94203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.556617] LNet: Removed LNI 192.168.206.9@tcp [ 1454.866716] Key type .llcrypt unregistered [ 1454.874252] Key type ._llcrypt unregistered [ 1463.356905] Key type ._llcrypt registered [ 1463.364362] Key type .llcrypt registered [ 1464.270408] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1465.591217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1466.867472] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:04:26 (1774875866) [ 1467.252151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1467.268485] alg: No test for adler32 (adler32-zlib) [ 1468.142387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.202132] LNet: 95289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.679576] Key type .llcrypt unregistered [ 1468.681258] Key type ._llcrypt unregistered [ 1473.537207] Key type ._llcrypt registered [ 1473.540076] Key type .llcrypt registered [ 1473.696481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1473.713082] alg: No test for adler32 (adler32-zlib) [ 1474.583615] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1474.588690] LNet: 95545:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1474.597320] LNet: 95545:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1474.625081] LNetError: 95557:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.11@tcp is being used as a gateway but routing feature is not turned on [ 1474.646323] LNetError: 95557:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.206.11@tcp has gone from up to down [ 1475.095730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1475.116691] LNet: 95603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.126995] LNetError: 95603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.153887] LNet: Removed LNI 192.168.206.9@tcp [ 1475.174183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.225381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1475.247910] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1475.253910] LNet: 95610:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1475.260427] LNet: 95610:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1475.265877] LNetError: 95610:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.206.10@tcp is a gateway. Can not delete it [ 1475.281424] LNetError: 95610:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.206.10@tcp is a router. Can not be deleted [ 1475.295125] LNetError: 95607:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.11@tcp is being used as a gateway but routing feature is not turned on [ 1475.311686] LNetError: 95607:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 1475.326446] LNetError: 95607:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.206.11@tcp has gone from up to down [ 1475.342795] LNetError: 95607:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1475.348372] LNet: 95610:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1480.401541] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:04:40 (1774875880) [ 1481.901413] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg609-server [ 1483.546330] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:04:43 (1774875883) [ 1484.873981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg609-server [ 1486.128822] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:04:46 (1774875886) [ 1487.332608] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 1489.111168] Key type lgssc registered [ 1490.032950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1505.872085] Key type lgssc unregistered [ 1506.140581] LNet: 98032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.145234] LNetError: 98032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.160877] LNet: Removed LNI 192.168.206.9@tcp [ 1506.656169] Key type .llcrypt unregistered [ 1506.660253] Key type ._llcrypt unregistered [ 1519.477203] Key type ._llcrypt registered [ 1519.479095] Key type .llcrypt registered [ 1520.533226] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:05:20 (1774875920) [ 1521.079849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1521.103782] alg: No test for adler32 (adler32-zlib) [ 1521.985500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.039754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1522.103231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1522.153405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1522.191846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1528.451579] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:05:27 (1774875927) [ 1528.529260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.549159] LNet: 99432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.576400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.651090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1528.684502] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1529.529392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1529.551539] LNet: 99582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.556490] LNetError: 99582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1529.574877] LNet: Removed LNI 192.168.206.9@tcp [ 1529.595354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.646685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1529.674314] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1530.200135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.221179] LNetError: 99610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.238970] LNet: Removed LNI 192.168.206.9@tcp [ 1530.272148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.339917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1530.367390] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1535.999671] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:05:35 (1774875935) [ 1536.115874] LNet: 100199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.120845] LNet: 100199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1536.127410] LNetError: 100199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.147564] LNet: Removed LNI 192.168.206.9@tcp [ 1536.704747] Key type .llcrypt unregistered [ 1536.708876] Key type ._llcrypt unregistered [ 1542.079651] Key type ._llcrypt registered [ 1542.082188] Key type .llcrypt registered [ 1542.316280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1542.337478] alg: No test for adler32 (adler32-zlib) [ 1543.284479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.353490] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1543.356465] LNet: *** cfs_fail_loc=e003, val=100*** [ 1543.965635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1544.001409] LNet: *** cfs_fail_loc=e003, val=100*** [ 1544.006211] LNet: Skipped 2 previous similar messages [ 1544.017755] Ping reply received with mapping: [ 1544.020679] found metadata reply NI[192.168.206.9@tcp] [ 1544.028591] 09 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1544.034724] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1544.040942] 5a 5a 5a 5a 5a 5a 5a 5a [ 1544.182776] LNet: 100600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.202995] LNetError: 100600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.233246] LNet: Removed LNI 192.168.206.9@tcp [ 1544.805519] Key type .llcrypt unregistered [ 1544.807643] Key type ._llcrypt unregistered [ 1556.039424] Key type ._llcrypt registered [ 1556.045347] Key type .llcrypt registered [ 1557.163577] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:05:57 (1774875957) [ 1557.763071] Key type .llcrypt unregistered [ 1557.765574] Key type ._llcrypt unregistered [ 1564.097490] Key type ._llcrypt registered [ 1564.100970] Key type .llcrypt registered [ 1564.277586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1564.300170] alg: No test for adler32 (adler32-zlib) [ 1565.424717] LNetError: 101646:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1565.862817] Key type .llcrypt unregistered [ 1565.866631] Key type ._llcrypt unregistered [ 1571.019104] Key type ._llcrypt registered [ 1571.020289] Key type .llcrypt registered [ 1572.178277] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:06:12 (1774875972) [ 1572.780499] Key type .llcrypt unregistered [ 1572.787624] Key type ._llcrypt unregistered [ 1579.652739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1579.660279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1582.228851] Key type ._llcrypt registered [ 1582.231501] Key type .llcrypt registered [ 1582.442280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1582.466396] alg: No test for adler32 (adler32-zlib) [ 1583.371993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.417720] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1591.471944] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:06:31 (1774875991) [ 1591.686787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.714073] LNet: 103428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.720711] LNetError: 103428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1591.745312] LNet: Removed LNI 10.1.2.3@tcp [ 1591.767405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.414679] Key type .llcrypt unregistered [ 1592.418157] Key type ._llcrypt unregistered [ 1597.527742] Key type ._llcrypt registered [ 1597.532210] Key type .llcrypt registered [ 1597.676312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1597.696796] alg: No test for adler32 (adler32-zlib) [ 1598.570932] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1598.617873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1598.743803] LNet: 103717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.749325] LNetError: 103717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.799501] LNet: Removed LNI 192.168.206.9@tcp [ 1600.366920] Key type .llcrypt unregistered [ 1600.369702] Key type ._llcrypt unregistered [ 1610.703911] Key type ._llcrypt registered [ 1610.712460] Key type .llcrypt registered [ 1611.820236] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:06:51 (1774876011) [ 1612.431555] Key type .llcrypt unregistered [ 1612.442401] Key type ._llcrypt unregistered [ 1617.864812] Key type ._llcrypt registered [ 1617.867710] Key type .llcrypt registered [ 1618.109496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1618.147891] alg: No test for adler32 (adler32-zlib) [ 1624.681581] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:07:04 (1774876024) [ 1624.832573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.893839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.992121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1625.012822] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1625.819957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1625.849547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1625.880952] LNet: Removed LNI 192.168.206.9@tcp [ 1626.125638] LNet: 105461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.494704] Key type .llcrypt unregistered [ 1626.497891] Key type ._llcrypt unregistered [ 1637.035278] Key type ._llcrypt registered [ 1637.037565] Key type .llcrypt registered [ 1638.415569] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:07:18 (1774876038) [ 1638.930531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1638.950223] alg: No test for adler32 (adler32-zlib) [ 1639.909889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.992959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.018760] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1640.564712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1640.590210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.9@tcp [ 1640.607170] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1640.610931] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1640.614762] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1640, 0, 0 [ 1640.618659] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1640.621206] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1640.625302] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1640.630195] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1640, 0, 0 [ 1640.636432] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1640.638619] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1640.641906] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1640.643555] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1640, 0, 1690 [ 1640.647625] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1640.650314] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1640.653666] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1640.656275] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1640, 0, 1690 [ 1640.663886] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1646.217591] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:07:26 (1774876046) [ 1646.664739] LNet: 107000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.669603] LNetError: 107000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.688092] LNet: Removed LNI 192.168.206.9@tcp [ 1647.258606] Key type .llcrypt unregistered [ 1647.267571] Key type ._llcrypt unregistered [ 1653.118335] Key type ._llcrypt registered [ 1653.122127] Key type .llcrypt registered [ 1653.307698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1653.328436] alg: No test for adler32 (adler32-zlib) [ 1654.307234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1654.354846] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1661.495549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1669.197838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.241926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1669.277462] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1670.158037] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1670.231883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1670.238766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1670.247383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1672.616513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1672.635989] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1682.442208] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:08:02 (1774876082) [ 1682.717544] LNet: 108509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.727956] LNetError: 108509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.736762] LNet: Removed LNI 192.168.206.9@tcp [ 1683.743852] LNet: Removed LNI 192.168.206.9@tcp1 [ 1683.747359] LNetError: 108509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1683.753322] LNetError: 108509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1683.773700] LNet: Removed LNI 10.1.2.3@tcp99 [ 1684.418791] Key type .llcrypt unregistered [ 1684.421232] Key type ._llcrypt unregistered [ 1690.446662] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1690.453675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1693.112650] Key type ._llcrypt registered [ 1693.115624] Key type .llcrypt registered [ 1693.260764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1693.273242] alg: No test for adler32 (adler32-zlib) [ 1694.198382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.245520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.276903] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1694.866796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1694.902114] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1697.182817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1697.234781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1697.293928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1697.338997] LNetError: 109254:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1697.374739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1697.430201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1698.053069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1698.142206] LNet: 109274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.146828] LNetError: 109274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.167195] LNet: Removed LNI 192.168.206.9@tcp [ 1699.176010] LNetError: 109274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1699.191889] LNet: Removed LNI 10.1.2.3@tcp [ 1699.895735] Key type .llcrypt unregistered [ 1699.898615] Key type ._llcrypt unregistered [ 1710.307988] Key type ._llcrypt registered [ 1710.311338] Key type .llcrypt registered [ 1711.436814] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:08:31 (1774876111) [ 1711.848431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1711.875080] alg: No test for adler32 (adler32-zlib) [ 1712.778865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.822728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.846233] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1713.464526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.9@tcp [ 1713.522526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg609-client@tcp [ 1719.756659] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:08:39 (1774876119) [ 1719.917601] LNet: 110806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.929182] LNetError: 110806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1721.001396] LNet: Removed LNI 192.168.206.9@tcp [ 1721.655625] Key type .llcrypt unregistered [ 1721.659196] Key type ._llcrypt unregistered [ 1727.585636] Key type ._llcrypt registered [ 1727.588388] Key type .llcrypt registered [ 1727.802614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1727.826156] alg: No test for adler32 (adler32-zlib) [ 1728.739471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1728.776531] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1728.826426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1735.813139] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:08:55 (1774876135) [ 1737.298302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1738.813942] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:08:58 (1774876138) [ 1740.138933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1741.481865] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:09:01 (1774876141) [ 1741.581641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.616292] LNet: 112220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.622153] LNetError: 112220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.644438] LNet: Removed LNI 192.168.206.9@tcp [ 1741.674580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.729473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.762203] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1742.356299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1742.390923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1748.199042] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:09:08 (1774876148) [ 1748.328697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.352179] LNet: 112930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.358421] LNetError: 112930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.395591] LNet: Removed LNI 192.168.206.9@tcp [ 1748.419458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.462536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1748.486177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1749.017356] Key type .llcrypt unregistered [ 1749.019158] Key type ._llcrypt unregistered [ 1758.210828] Key type ._llcrypt registered [ 1758.219380] Key type .llcrypt registered [ 1759.623169] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:09:19 (1774876159) [ 1759.996287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1760.011615] alg: No test for adler32 (adler32-zlib) [ 1760.933863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.990917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.007706] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1761.821318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1761.902457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1767.877263] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:09:27 (1774876167) [ 1767.955210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.990391] LNet: 114488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.996175] LNetError: 114488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.011601] LNet: Removed LNI 192.168.206.9@tcp [ 1768.052669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.112701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1768.171781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1773.581868] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:09:33 (1774876173) [ 1773.672992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.695075] LNet: 115068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.715116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.793542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1774.202941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1774.239488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1774.305384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1774.717882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1774.748183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1774.823788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.846465] LNet: 115107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.870467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.935443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1775.288780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1775.335374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1775.397370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1775.861329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1775.907888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1781.818301] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:09:41 (1774876181) [ 1782.000928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.019660] LNet: 115707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.044383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.100687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.131180] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1782.867261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1782.906426] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1783.829660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1783.903506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1784.141915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.164468] LNetError: 115986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.183330] LNet: Removed LNI 192.168.206.9@tcp [ 1785.191340] LNetError: 115986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.215503] LNet: Removed LNI 192.168.206.9@tcp2 [ 1785.247469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.315953] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1785.327264] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1785.334796] LNet: 115992:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1786.003841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1786.265325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.291349] LNet: 116030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1786.297625] LNet: 116030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1787.359251] LNet: Removed LNI 192.168.206.9@tcp [ 1787.365226] LNetError: 116030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.376964] LNetError: 116030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1787.406971] LNet: Removed LNI 192.168.206.9@tcp2 [ 1787.462195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.533967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1787.572376] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1788.214090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1788.234086] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1788.937223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1788.983565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1789.036837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1789.225351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1790.303195] LNet: Removed LNI 192.168.206.9@tcp [ 1791.333382] LNet: Removed LNI 192.168.206.9@tcp2 [ 1791.366369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.453350] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1791.462364] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1791.467792] LNet: 116320:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1792.084359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1792.340301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.372892] LNetError: 116357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1792.384189] LNetError: 116357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1792.393833] LNet: Removed LNI 192.168.206.9@tcp [ 1793.449996] LNet: Removed LNI 192.168.206.9@tcp2 [ 1793.501928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.579550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1793.612542] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1794.227899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1794.250900] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1794.959280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1795.009542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1795.191488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.220167] LNet: 116637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.229300] LNet: 116637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1796.256520] LNet: Removed LNI 192.168.206.9@tcp [ 1796.268827] LNet: Removed LNI 192.168.206.9@tcp2 [ 1796.299401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.355452] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1796.363948] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1796.369975] LNet: 116643:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1796.828353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1797.013495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.047210] LNet: Removed LNI 192.168.206.9@tcp [ 1798.058667] LNet: Removed LNI 192.168.206.9@tcp2 [ 1798.081413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.145273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.171720] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1798.759172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1798.775636] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1799.444715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1799.473540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1799.519825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1799.669455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.735178] LNet: Removed LNI 192.168.206.9@tcp [ 1800.738566] LNetError: 116964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.743288] LNetError: 116964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1800.753838] LNet: Removed LNI 192.168.206.9@tcp2 [ 1800.774844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.838100] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1800.848485] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1800.852559] LNet: 116971:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1801.195347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1801.367268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.389599] LNet: Removed LNI 192.168.206.9@tcp [ 1802.412432] LNet: Removed LNI 192.168.206.9@tcp2 [ 1802.444329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.492272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.522736] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1803.223736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1803.245847] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1804.065722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1804.095946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1804.151800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1804.575709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.599597] LNet: Removed LNI 192.168.206.9@tcp [ 1805.611342] LNet: Removed LNI 192.168.206.9@tcp2 [ 1805.643729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.703147] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1805.711589] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1805.718624] LNet: 117310:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1806.251711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1806.592846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.647234] LNet: Removed LNI 192.168.206.9@tcp [ 1807.659894] LNet: Removed LNI 192.168.206.9@tcp2 [ 1807.677183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.725980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1807.748167] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1808.296139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1808.312984] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1809.209652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1809.261555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1809.290952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1809.356600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1809.735172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1810.784921] LNet: Removed LNI 192.168.206.9@tcp [ 1811.813790] LNet: Removed LNI 192.168.206.9@tcp2 [ 1811.832789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.900754] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1811.906354] LNet: Added LNI 192.168.206.9@tcp2 [8/256/0/180] [ 1812.536130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1818.235804] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1819.515650] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:10:19 (1774876219) [ 1819.602802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.648267] LNet: 118422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.652356] LNet: 118422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1819.657648] LNetError: 118422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.662252] LNetError: 118422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1819.672450] LNet: Removed LNI 192.168.206.9@tcp [ 1819.693668] LNet: Removed LNI 192.168.206.9@tcp2 [ 1819.730528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1819.985856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1820.008348] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1820.658491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1820.675828] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1821.527560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.545437] LNet: Removed LNI 192.168.206.9@tcp [ 1821.554663] LNet: Removed LNI 10.1.2.3@tcp [ 1821.571635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.619531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1821.639273] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1821.643092] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1821.994890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1822.021919] LNet: Removed LNI 192.168.206.9@tcp [ 1822.039592] LNet: Removed LNI 10.1.2.3@tcp [ 1828.115607] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:10:27 (1774876227) [ 1828.217903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.293832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.439121] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1828.642332] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1828.645895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1828.651418] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1830.898847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1830.925747] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1831.556293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1831.574360] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1833.628482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.9@tcp [ 1833.694335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1833.755299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.9@tcp -m PUT -e local_timeout -r 1 [ 1833.814626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1833.873630] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1833.878650] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1834.096091] LNetError: 119369:0:(lib-move.c:4969:lnet_parse()) 192.168.206.9@tcp, src 192.168.206.9@tcp: Dropping PUT (error -108 looking up sender) [ 1836.128054] LNet: Removed LNI 192.168.206.9@tcp [ 1836.138814] LNet: Removed LNI 10.1.2.3@tcp [ 1836.524739] Key type .llcrypt unregistered [ 1836.528779] Key type ._llcrypt unregistered [ 1844.989517] Key type ._llcrypt registered [ 1844.991482] Key type .llcrypt registered [ 1846.261850] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:10:45 (1774876245) [ 1846.601206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.623068] alg: No test for adler32 (adler32-zlib) [ 1847.496350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.932465] LNet: 120679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1848.551570] Key type .llcrypt unregistered [ 1848.557347] Key type ._llcrypt unregistered [ 1854.206959] Key type ._llcrypt registered [ 1854.214486] Key type .llcrypt registered [ 1854.522531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1854.549684] alg: No test for adler32 (adler32-zlib) [ 1855.477560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1855.504774] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1861.896196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1862.126811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1862.713187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.109@tcp [ 1863.012370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1863.065534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1863.127908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1863.176617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1863.226537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1863.273628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1863.304871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1863.342381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1863.363539] LNet: 121128:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1863.379992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1863.422196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1863.475553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1863.514340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1863.554803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1863.588522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1864.135599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1864.325557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1864.356598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1864.395731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1864.430763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1864.455868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1864.503323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1864.532911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1864.566447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1864.585230] LNet: 121245:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1864.602446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1864.629423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1864.657446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1864.688613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1864.715356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1864.743856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1865.260731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.109@tcp [ 1865.456181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1865.494328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1865.520918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1865.542298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1865.569345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1865.595222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1865.619648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1865.645563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1865.660802] LNet: 121362:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1865.677958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1865.705866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1865.737533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1865.759521] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.109@tcp [ 1865.765716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1865.801465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1865.846607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1866.389263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1866.567220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1866.609897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1866.655435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1866.692260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1866.717746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1866.752505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1866.784412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1866.814718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1866.855626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1866.907576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1866.936747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1866.966943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1866.993992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1867.024970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1867.291281] LNet: 121526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.297327] LNetError: 121526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.320452] LNet: Removed LNI 192.168.206.9@tcp [ 1867.696441] Key type .llcrypt unregistered [ 1867.703057] Key type ._llcrypt unregistered [ 1881.755463] Key type ._llcrypt registered [ 1881.757777] Key type .llcrypt registered [ 1882.913931] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:11:22 (1774876282) [ 1883.255071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1883.273815] alg: No test for adler32 (adler32-zlib) [ 1884.178916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.573694] LNet: 122412:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.191479] Key type .llcrypt unregistered [ 1885.195454] Key type ._llcrypt unregistered [ 1890.534872] Key type ._llcrypt registered [ 1890.542046] Key type .llcrypt registered [ 1890.738933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1890.757317] alg: No test for adler32 (adler32-zlib) [ 1891.678668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1891.701059] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 1898.799804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.109@tcp [ 1907.473378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.530777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1907.562591] LNet: Added LNI 192.168.206.9@tcp1 [8/256/0/180] [ 1908.526617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1909.337607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.109@tcp [ 1909.686731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1909.714944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1909.755416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1909.797254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1909.855441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1909.883900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1909.911173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1909.938756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1909.967773] LNet: 123064:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1910.003905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1910.050666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1910.097438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1910.143361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1910.176223] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.109@tcp [ 1910.179904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1910.219796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1910.813794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1911.070080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1911.130081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1911.158868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1911.184312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1911.216694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1911.255192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1911.282827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1911.319321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1911.337282] LNet: 123181:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1911.366799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1911.409929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1911.453661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1911.486930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1911.546362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1911.578134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1912.190124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.109@tcp [ 1912.414910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1912.438945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1912.471747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1912.515992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1912.554089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1912.604592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1912.655795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1912.694579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1912.726914] LNet: 123298:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1912.773926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1912.848134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1912.888361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1912.922163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1912.982192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1913.049942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1913.719949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1913.946612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1913.980955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1914.006434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1914.037277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1914.064981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1914.094429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1914.150370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1914.208659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1914.250423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1914.288825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1914.333334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1914.378652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1914.431775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1914.505537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1919.311641] LNet: 123496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.316972] LNetError: 123496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.326042] LNet: Removed LNI 192.168.206.9@tcp [ 1919.340312] LNet: Removed LNI 192.168.206.9@tcp1 [ 1919.820877] Key type .llcrypt unregistered [ 1919.823239] Key type ._llcrypt unregistered [ 1935.880975] Key type ._llcrypt registered [ 1935.890757] Key type .llcrypt registered [ 1937.212765] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:12:16 (1774876336) [ 1945.582829] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:12:25 (1774876345) [ 1953.465063] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:12:33 (1774876353) [ 1961.056866] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:12:40 (1774876360) [ 1967.990564] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:12:47 (1774876367) [ 1975.752986] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:12:55 (1774876375) [ 1982.878366] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:13:02 (1774876382) [ 1989.318349] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:13:09 (1774876389) [ 1996.367196] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:13:15 (1774876395) [ 2003.437259] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:13:23 (1774876403) [ 2010.188487] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:13:30 (1774876410) [ 2017.316332] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:13:37 (1774876417) [ 2023.941818] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:13:43 (1774876423) [ 2029.876884] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:13:49 (1774876429) [ 2037.169200] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:13:56 (1774876436) [ 2043.930340] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:14:03 (1774876443) [ 2050.421683] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:14:10 (1774876450) [ 2056.890884] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:14:16 (1774876456) [ 2057.403849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.418301] alg: No test for adler32 (adler32-zlib) [ 2058.313598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.436840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2058.454095] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2059.573744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.594965] LNet: 134206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.601279] LNetError: 134206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.616061] LNet: Removed LNI 192.168.206.9@tcp [ 2059.639375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.700147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2059.715846] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2066.265635] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:14:25 (1774876465) [ 2073.933686] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:14:33 (1774876473) [ 2080.970652] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:14:40 (1774876480) [ 2087.590664] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:14:47 (1774876487) [ 2094.405662] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:14:54 (1774876494) [ 2100.361582] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:15:00 (1774876500) [ 2106.476312] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:15:06 (1774876506) [ 2113.315742] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:15:13 (1774876513) [ 2119.239081] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:15:19 (1774876519) [ 2124.594753] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:15:24 (1774876524) [ 2131.160746] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:15:30 (1774876530) [ 2137.371994] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:15:37 (1774876537) [ 2144.085160] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1952 sec ========== 09:15:43 (1774876543) [ 2144.456515] LNet: 141672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.462760] LNetError: 141672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.484861] LNet: Removed LNI 192.168.206.9@tcp [ 2145.208585] Key type .llcrypt unregistered [ 2145.213201] Key type ._llcrypt unregistered [ 2155.773406] Key type ._llcrypt registered [ 2155.775770] Key type .llcrypt registered [ 2155.988758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2156.000957] alg: No test for adler32 (adler32-zlib) [ 2157.041803] Lustre: Lustre: Build Version: 2.17.51_23_gefc2bdf [ 2157.259489] LNet: Added LNI 192.168.206.9@tcp [8/256/0/180] [ 2158.927316] Key type lgssc registered [ 2159.965260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2206.114271] Lustre: Mounted lustre-client [ 2210.601784] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2221.998337] LustreError: 144126:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8e7f145cc000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2222.056080] Lustre: Unmounted lustre-client [ 2238.593965] Key type lgssc unregistered [ 2238.827740] LNet: 144607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.832630] LNetError: 144607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.856323] LNet: Removed LNI 192.168.206.9@tcp [ 2239.445163] Key type .llcrypt unregistered [ 2239.448127] Key type ._llcrypt unregistered