[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 424258554 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003118] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021116] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026654] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030111] *** VALIDATE tmpfs *** [ 0.031463] *** VALIDATE proc *** [ 0.033226] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035298] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039010] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043235] debug: unmapping init [mem 0xffffffffa9259000-0xffffffffa9260fff] [ 0.045178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046698] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048012] ... bit width: 48 [ 0.049011] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052016] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054299] rcu: Hierarchical SRCU implementation. [ 0.056459] smp: Bringing up secondary CPUs ... [ 0.057468] x86: Booting SMP configuration: [ 0.058021] .... node #0, CPUs: #1 #2 #3 [ 0.061715] smp: Brought up 1 node, 4 CPUs [ 0.063010] smpboot: Max logical packages: 1 [ 0.063977] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.153866] node 0 deferred pages initialised in 88ms [ 0.157111] devtmpfs: initialized [ 0.158225] x86/mm: Memory block size: 128MB [ 0.161129] gcov: version magic: 0x41383552 [ 0.162733] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.168197] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.171301] pinctrl core: initialized pinctrl subsystem [ 0.173190] [ 0.174037] ************************************************************* [ 0.177060] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.180014] ** ** [ 0.182011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.185016] ** ** [ 0.188012] ** This means that this kernel is built to expose internal ** [ 0.190011] ** IOMMU data structures, which may compromise security on ** [ 0.193013] ** your system. ** [ 0.196013] ** ** [ 0.198011] ** If you see this message and you are not debugging the ** [ 0.201014] ** kernel, report this immediately to your vendor! ** [ 0.203011] ** ** [ 0.206013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.208022] ************************************************************* [ 0.210636] NET: Registered protocol family 16 [ 0.213436] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.216058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.218042] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.222031] cpuidle: using governor menu [ 0.224000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.226430] PCI: Using configuration type 1 for base access [ 0.229146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.239089] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.241032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.245059] cryptd: max_cpu_qlen set to 1000 [ 0.247236] ACPI: Added _OSI(Module Device) [ 0.249015] ACPI: Added _OSI(Processor Device) [ 0.250012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.251010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.256603] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.261542] ACPI: Interpreter enabled [ 0.263059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.264014] ACPI: Using IOAPIC for interrupt routing [ 0.266116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.270368] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.280443] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.283038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.285019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.288101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293444] acpiphp: Slot [2] registered [ 0.294113] acpiphp: Slot [5] registered [ 0.296136] acpiphp: Slot [6] registered [ 0.298111] acpiphp: Slot [3] registered [ 0.299115] acpiphp: Slot [4] registered [ 0.300107] acpiphp: Slot [7] registered [ 0.302109] acpiphp: Slot [8] registered [ 0.304098] acpiphp: Slot [9] registered [ 0.306094] acpiphp: Slot [10] registered [ 0.307110] acpiphp: Slot [11] registered [ 0.309113] acpiphp: Slot [12] registered [ 0.310085] acpiphp: Slot [13] registered [ 0.312148] acpiphp: Slot [14] registered [ 0.313091] acpiphp: Slot [15] registered [ 0.315111] acpiphp: Slot [16] registered [ 0.316091] acpiphp: Slot [17] registered [ 0.318091] acpiphp: Slot [18] registered [ 0.319095] acpiphp: Slot [19] registered [ 0.320090] acpiphp: Slot [20] registered [ 0.323121] acpiphp: Slot [21] registered [ 0.324167] acpiphp: Slot [22] registered [ 0.326097] acpiphp: Slot [23] registered [ 0.327117] acpiphp: Slot [24] registered [ 0.329101] acpiphp: Slot [25] registered [ 0.330093] acpiphp: Slot [26] registered [ 0.332100] acpiphp: Slot [27] registered [ 0.334082] acpiphp: Slot [28] registered [ 0.336115] acpiphp: Slot [29] registered [ 0.337103] acpiphp: Slot [30] registered [ 0.339124] acpiphp: Slot [31] registered [ 0.341067] PCI host bridge to bus 0000:00 [ 0.342018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.344018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.346022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.349024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.352021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.355049] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.357157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.359885] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.363328] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.371014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.375072] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.378018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.381017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.383020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.387123] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.389883] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.391063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.394857] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.399013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.410013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419701] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.446017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.456641] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.466015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.473016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.493019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.502503] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.505350] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.507296] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.509381] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.511195] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.515111] iommu: Default domain type: Passthrough [ 0.517328] SCSI subsystem initialized [ 0.519128] ACPI: bus type USB registered [ 0.520105] usbcore: registered new interface driver usbfs [ 0.522067] usbcore: registered new interface driver hub [ 0.523074] usbcore: registered new device driver usb [ 0.525157] pps_core: LinuxPPS API ver. 1 registered [ 0.526009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.528056] PTP clock support registered [ 0.531046] EDAC MC: Ver: 3.0.0 [ 0.532536] PCI: Using ACPI for IRQ routing [ 0.534767] NetLabel: Initializing [ 0.536010] NetLabel: domain hash size = 128 [ 0.537010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.539078] NetLabel: unlabeled traffic allowed by default [ 0.541134] vgaarb: loaded [ 0.543253] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.545014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.555000] clocksource: Switched to clocksource kvm-clock [ 0.664809] VFS: Disk quotas dquot_6.6.0 [ 0.666579] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.669144] *** VALIDATE ramfs *** [ 0.670279] *** VALIDATE hugetlbfs *** [ 0.671902] pnp: PnP ACPI init [ 0.674796] pnp: PnP ACPI: found 6 devices [ 0.693135] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.695881] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.697648] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.699555] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.701580] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.703466] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.705713] NET: Registered protocol family 2 [ 0.707929] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.712995] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.716228] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.722540] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.725336] TCP: Hash tables configured (established 65536 bind 65536) [ 0.728503] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.732278] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.735301] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.739749] NET: Registered protocol family 1 [ 0.743804] RPC: Registered named UNIX socket transport module. [ 0.746399] RPC: Registered udp transport module. [ 0.748458] RPC: Registered tcp transport module. [ 0.749953] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.752110] NET: Registered protocol family 44 [ 0.753686] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.755577] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.757297] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.758871] PCI: CLS 0 bytes, default 64 [ 0.760348] Unpacking initramfs... [ 2.213450] debug: unmapping init [mem 0xffff982ebcc64000-0xffff982ebffcffff] [ 2.219916] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.222186] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.225969] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.724122] Initialise system trusted keyrings [ 2.725877] Key type blacklist registered [ 2.727775] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.737268] zbud: loaded [ 2.740535] *** VALIDATE nfs *** [ 2.741781] *** VALIDATE nfs4 *** [ 2.744393] pstore: using deflate compression [ 2.748300] Platform Keyring initialized [ 2.854570] NET: Registered protocol family 38 [ 2.856634] Key type asymmetric registered [ 2.858194] Asymmetric key parser 'x509' registered [ 2.859785] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.862776] io scheduler mq-deadline registered [ 2.864520] io scheduler kyber registered [ 2.866212] io scheduler bfq registered [ 2.868130] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.870827] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.873919] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.876571] ACPI: Power Button [PWRF] [ 2.882527] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.889760] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.904307] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.932330] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.962973] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.968448] Non-volatile memory driver v1.3 [ 2.970315] Linux agpgart interface v0.103 [ 3.011581] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.014212] vda: detected capacity change from 0 to 74702848 [ 3.031393] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.035316] vdb: detected capacity change from 0 to 1073741824 [ 3.042256] libphy: Fixed MDIO Bus: probed [ 3.059815] usbcore: registered new interface driver usbserial_generic [ 3.062742] usbserial: USB Serial support registered for generic [ 3.066106] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.071844] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.074070] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.076838] mousedev: PS/2 mouse device common for all mice [ 3.080160] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.082896] rtc_cmos 00:05: RTC can wake from S4 [ 3.085825] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.088500] rtc_cmos 00:05: registered as rtc0 [ 3.091177] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.091365] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.097624] intel_pstate: CPU model not supported [ 3.101470] hid: raw HID events driver (C) Jiri Kosina [ 3.104073] usbcore: registered new interface driver usbhid [ 3.106438] usbhid: USB HID core driver [ 3.108561] drop_monitor: Initializing network drop monitor service [ 3.111551] Initializing XFRM netlink socket [ 3.114118] NET: Registered protocol family 10 [ 3.117438] Segment Routing with IPv6 [ 3.118654] NET: Registered protocol family 17 [ 3.120398] mpls_gso: MPLS GSO support [ 3.125711] RAS: Correctable Errors collector initialized. [ 3.127710] AVX version of gcm_enc/dec engaged. [ 3.129227] AES CTR mode by8 optimization enabled [ 3.218306] sched_clock: Marking stable (3218282005, 0)->(4122426590, -904144585) [ 3.222299] registered taskstats version 1 [ 3.225201] Loading compiled-in X.509 certificates [ 3.227093] zswap: loaded using pool lzo/zbud [ 3.251453] Key type big_key registered [ 3.263510] Key type encrypted registered [ 3.265485] ima: No TPM chip found, activating TPM-bypass! [ 3.267915] ima: Allocated hash algorithm: sha1 [ 3.270281] ima: No architecture policies found [ 3.272500] evm: Initialising EVM extended attributes: [ 3.274333] evm: security.selinux [ 3.275620] evm: security.ima [ 3.276776] evm: security.capability [ 3.278174] evm: HMAC attrs: 0x1 [ 3.280446] rtc_cmos 00:05: setting system clock to 2026-08-15 20:40:34 UTC (1786826434) [ 3.288553] debug: unmapping init [mem 0xffffffffaa203000-0xffffffffaa3fffff] [ 3.291730] debug: unmapping init [mem 0xffffffffa8f82000-0xffffffffa9258fff] [ 3.301094] Write protecting the kernel read-only data: 28672k [ 3.306489] debug: unmapping init [mem 0xffffffffa7603000-0xffffffffa77fffff] [ 3.309882] debug: unmapping init [mem 0xffffffffa7f14000-0xffffffffa7ffffff] [ 3.347064] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.356456] systemd[1]: Detected virtualization kvm. [ 3.358577] systemd[1]: Detected architecture x86-64. [ 3.360610] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.391096] systemd[1]: No hostname configured. [ 3.393071] systemd[1]: Set hostname to . [ 3.395319] random: systemd: uninitialized urandom read (16 bytes read) [ 3.397660] systemd[1]: Initializing machine ID from random generator. [ 3.442217] random: ln: uninitialized urandom read (6 bytes read) [ 3.524699] random: systemd: uninitialized urandom read (16 bytes read) [ 3.527725] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.533474] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.538784] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.136742] device-mapper: uevent: version 1.0.3 [ 4.139537] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.872346] virtio_net virtio0 ens2: renamed from eth0 [ 4.902369] random: fast init done [ 4.924494] scsi host0: ata_piix [ 4.936661] scsi host1: ata_piix [ 4.938608] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.941443] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.605050] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.792296] random: crng init done [ 9.793811] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.146392] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.217288] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.470461] SELinux: Disabled at runtime. [ 11.531546] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.541334] systemd[1]: Detected virtualization kvm. [ 11.543419] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.000421] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.004161] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.011367] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.014323] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.016833] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.022944] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.030810] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK [[ 12.135642] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.415177] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.746843] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.761564] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.871199] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.881331] EDAC sbridge: Ver: 1.1.2 [ 13.852401] Key type dns_resolver registered [ 14.154744] NFS: Registering the id_resolver key type [ 14.157108] Key type id_resolver registered [ 14.158656] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg616-client login: [ 43.864898] libcfs: loading out-of-tree module taints kernel. [ 43.927184] Key type ._llcrypt registered [ 43.933095] Key type .llcrypt registered [ 44.439205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 44.474402] alg: No test for adler32 (adler32-zlib) [ 44.829086] hrtimer: interrupt took 2076233 ns [ 45.888944] Lustre: Lustre: Build Version: 2.17.57_4_gbf75d46 [ 46.917887] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 48.688321] Key type lgssc registered [ 50.363951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 161.230701] Lustre: Mounted lustre-client - version 2.17.57_4_gbf75d46 [ 166.399909] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 180.360239] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing check_logdir /tmp/testlogs/ [ 185.752888] Lustre: DEBUG MARKER: oleg616-client.virtnet: executing yml_node [ 186.851545] Lustre: lustre-OST0000-osc-ffff982f02d59000: disconnect after 24s idle [ 190.305288] Lustre: DEBUG MARKER: Client: 2.17.57.4 [ 192.511375] Lustre: DEBUG MARKER: MDS: 2.17.57.4 [ 194.969262] Lustre: DEBUG MARKER: OSS: 2.17.57.4 [ 196.679331] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 15 16:43:46 EDT 2026 [ 210.761145] Lustre: Unmounted lustre-client [ 239.293941] Key type lgssc unregistered [ 239.717909] LNet: 8411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 239.729440] LNetError: 8411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 239.763092] LNet: Removed LNI 192.168.206.16@tcp [ 240.658703] Key type .llcrypt unregistered [ 240.665670] Key type ._llcrypt unregistered [ 267.866400] Key type ._llcrypt registered [ 267.870377] Key type .llcrypt registered [ 268.184359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 268.212841] alg: No test for adler32 (adler32-zlib) [ 269.129987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 269.173248] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 269.190344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 272.935505] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 273.059102] LNet: 9292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 273.070668] LNetError: 9292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 273.094784] LNet: Removed LNI 192.168.206.16@tcp [ 273.810607] Key type .llcrypt unregistered [ 273.813149] Key type ._llcrypt unregistered [ 280.367586] Key type ._llcrypt registered [ 280.370944] Key type .llcrypt registered [ 281.494387] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:45:11 (1786826711) [ 282.012895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 282.023595] alg: No test for adler32 (adler32-zlib) [ 282.911906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.972247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.990196] LNet: 9714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.012763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.071596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 283.092735] LNet: 9721:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 288.292135] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:45:18 (1786826718) [ 288.607339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.635286] LNet: 10299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.664786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.778665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 294.813225] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:45:24 (1786826724) [ 294.931718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.961739] LNet: 10880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.987475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.074297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 301.322261] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:45:31 (1786826731) [ 301.542393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.595720] LNet: 11468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.631097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.715230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 307.754989] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:45:37 (1786826737) [ 307.866125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.901403] LNet: 12046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.956841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.078305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 308.890435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 315.794905] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:45:45 (1786826745) [ 315.880166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.912289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.041212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 316.531552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 316.861926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 323.819891] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:45:53 (1786826753) [ 324.277771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.309117] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.314682] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 324.339366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.460994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 331.099525] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:46:00 (1786826760) [ 331.311542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.394959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.494260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 331.551929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 332.038339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 332.076516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 332.502685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 332.552332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 332.930205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 332.967647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 333.403376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 333.477743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 333.793820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 333.817514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 334.160234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 334.181362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 334.628306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 334.664804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 335.109291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 335.153967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 335.556094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 335.594417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 336.105791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 336.136200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 336.509990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 336.551090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 337.088936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 337.132724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 342.730239] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:46:12 (1786826772) [ 342.874948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.895407] LNet: 14504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.901549] LNet: 14504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 342.928386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.002888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 343.033657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 349.703426] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:46:19 (1786826779) [ 349.803745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.885913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.992975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 350.043229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 357.323408] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:46:27 (1786826787) [ 357.522139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.628141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.727856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 357.769397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 365.459904] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:46:34 (1786826794) [ 365.603299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.693893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.790105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 365.829038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 372.888540] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:46:42 (1786826802) [ 372.973690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.997110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.072097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 373.102600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 381.205406] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:46:50 (1786826810) [ 381.293144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.324916] LNet: 17438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.329046] LNet: 17438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 381.365226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.439696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 381.485629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 388.645448] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:46:58 (1786826818) [ 388.793973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.906778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.090950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 389.635369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 396.210317] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:47:06 (1786826826) [ 396.263067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.284880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.462666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 396.689868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 402.846908] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:47:12 (1786826832) [ 403.112085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.190676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.346290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 403.627881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 409.830376] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:47:19 (1786826839) [ 409.944701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.983553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.507407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 410.853864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 416.635458] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:47:26 (1786826846) [ 416.841158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.914368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.031668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.103904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 423.303955] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:47:33 (1786826853) [ 423.540716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.606770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.680077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 423.733074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 429.331654] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:47:39 (1786826859) [ 429.452189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.492287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.576579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 435.517608] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:47:45 (1786826865) [ 435.604259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.647822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.757745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 441.616912] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:47:51 (1786826871) [ 441.705911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.734074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.803476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 442.073321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 448.141648] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:47:58 (1786826878) [ 448.387387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.409387] LNet: 23470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.415994] LNet: 23470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 448.450497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.551971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 448.597557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 454.681224] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:48:04 (1786826884) [ 454.941272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.008728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.134121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 455.195537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 461.098351] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:48:11 (1786826891) [ 461.474932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.545597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.599324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 461.635270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 468.397697] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:48:18 (1786826898) [ 468.584256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.660705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.740907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 468.777740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 474.586744] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:48:24 (1786826904) [ 474.746869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.829493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.900632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 474.969227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 474.991896] LNetError: 25827:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 475.030552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 475.065494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 481.092151] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:48:30 (1786826910) [ 481.373488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.444942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.554332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QKa8t75nun [ 481.645697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QKa8t75nun [ 481.758520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.QKa8t75nun [ 488.043427] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:48:38 (1786826918) [ 488.171978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.211323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.293530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 488.333373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 488.370245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 494.699597] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 16:48:44 (1786826924) [ 494.766987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.801729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.883676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 494.901900] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 496.563977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 497.881609] Lustre: Lustre: Build Version: 2.17.57_4_gbf75d46 [ 499.920320] Key type lgssc registered [ 501.332420] Lustre: Echo OBD driver; http://www.lustre.org/ [ 505.132585] LNetError: 27587:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.16@tcp99 has no available nets [ 507.699791] Key type lgssc unregistered [ 507.979171] LNetError: 28699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 508.004479] LNet: Removed LNI 192.168.206.16@tcp [ 508.518713] Key type .llcrypt unregistered [ 508.522604] Key type ._llcrypt unregistered [ 514.895727] Key type ._llcrypt registered [ 514.897144] Key type .llcrypt registered [ 516.285831] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:49:05 (1786826945) [ 516.789369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 516.809347] alg: No test for adler32 (adler32-zlib) [ 517.746123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.811201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 521.010255] LNet: Message forwarding will be enabled in 2s [ 524.064139] LNet: Message forwarding enabled [ 524.634942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 524.657512] LNet: Message forwarding will stop in 2s to 6s [ 529.197542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 529.410556] LNet: Message forwarding disabled [ 531.151352] LNet: Message forwarding will be enabled in 2s [ 534.561524] LNet: Message forwarding enabled [ 537.218413] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:49:27 (1786826967) [ 537.299353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.333726] LNet: 30102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 538.441234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.553288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 543.675833] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:49:33 (1786826973) [ 543.750601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.772463] LNet: 30681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.817850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.902950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 543.950131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 543.995246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 544.030279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 544.080306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 544.117821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 544.137640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 544.168172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 544.210059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 544.245230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 544.283364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 544.322671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 544.358394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 544.406926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 544.445754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 544.488286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 544.541595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 544.580846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 544.615716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 544.657735] LNetError: 30732:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 550.395974] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:49:40 (1786826980) [ 550.550595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.590230] LNet: 31299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.650695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.761832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 557.264104] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:49:47 (1786826987) [ 559.018973] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg616-server [ 560.740435] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:49:50 (1786826990) [ 560.821610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.853515] LNet: 32227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.875756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.935618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 560.957440] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 561.612476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[17-21]@tcp [ 561.626326] LNet: 32366:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 561.656929] LNetError: 32367:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.20@tcp has gone from up to down [ 561.679800] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.17@tcp is being used as a gateway but routing feature is not turned on [ 562.672655] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.17@tcp is being used as a gateway but routing feature is not turned on [ 562.693580] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 563.689861] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.17@tcp is being used as a gateway but routing feature is not turned on [ 563.704324] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 565.735356] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.21@tcp is being used as a gateway but routing feature is not turned on [ 565.744203] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 567.150762] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:49:57 (1786826997) [ 568.627695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg616-server [ 569.854372] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.21@tcp is being used as a gateway but routing feature is not turned on [ 569.870571] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 570.010764] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:50:00 (1786827000) [ 570.103361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.125371] LNet: 33299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.135532] LNetError: 33299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 571.174436] LNet: Removed LNI 192.168.206.16@tcp [ 571.238803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.290799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 571.329395] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 571.843515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[17-22/2]@tcp [ 571.862641] LNet: 33419:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 571.868548] LNet: 33419:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 571.886894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[17-22/2]@tcp [ 571.892787] LNetError: 33306:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.17@tcp has gone from up to down [ 571.909121] LNetError: 33306:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 577.544748] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:50:07 (1786827007) [ 577.719230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.735225] LNetError: 34011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 577.751409] LNet: Removed LNI 192.168.206.16@tcp [ 577.769973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.867635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 577.919168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.953909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.021979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 578.063590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.108769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.162197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 578.239653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.257537] LNet: 34051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 578.261915] LNet: 34051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 578.283332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.349877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 578.430632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.463727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.508691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 578.578378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.605960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.657335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 578.744566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.791881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.863788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 578.946383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.977779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.030653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 579.094438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.135514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.180794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 579.270123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.318362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.404384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 579.495989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.522911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.576468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 579.628564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.668514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.746339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 585.773690] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:50:15 (1786827015) [ 587.285637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg616-server [ 588.815157] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:50:18 (1786827018) [ 590.518696] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg616-server [ 591.886928] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:50:21 (1786827021) [ 592.907739] Key type .llcrypt unregistered [ 592.910159] Key type ._llcrypt unregistered [ 598.825579] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 598.880865] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 598.884163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 598.887945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 601.603219] Key type ._llcrypt registered [ 601.608571] Key type .llcrypt registered [ 601.864726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 601.903257] alg: No test for adler32 (adler32-zlib) [ 602.954854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.046706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 603.079809] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 603.771797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 603.794143] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 605.648798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 605.680135] LNet: Removed LNI 10.1.2.3@tcp [ 605.831803] LNet: 36139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.840197] LNetError: 36139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.853864] LNet: Removed LNI 192.168.206.16@tcp [ 606.514672] Key type .llcrypt unregistered [ 606.517524] Key type ._llcrypt unregistered [ 617.477112] Key type ._llcrypt registered [ 617.480580] Key type .llcrypt registered [ 619.220900] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:50:48 (1786827048) [ 619.867277] Key type .llcrypt unregistered [ 619.868988] Key type ._llcrypt unregistered [ 625.349808] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 625.503206] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 625.512580] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 625.521382] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 628.345359] Key type ._llcrypt registered [ 628.352413] Key type .llcrypt registered [ 628.602972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 628.613942] alg: No test for adler32 (adler32-zlib) [ 629.605361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.684091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 629.713549] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 630.386436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 630.402897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 632.616470] LNet: 37635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.637572] LNetError: 37635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 632.649606] LNet: Removed LNI 10.1.2.3@tcp [ 632.667517] LNet: Removed LNI 192.168.206.16@tcp [ 633.305630] Key type .llcrypt unregistered [ 633.309768] Key type ._llcrypt unregistered [ 642.950307] Key type ._llcrypt registered [ 642.953256] Key type .llcrypt registered [ 643.873223] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:51:14 (1786827074) [ 644.414490] Key type .llcrypt unregistered [ 644.417066] Key type ._llcrypt unregistered [ 650.404829] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 650.586278] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 650.599900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 650.608756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 653.193947] Key type ._llcrypt registered [ 653.196182] Key type .llcrypt registered [ 653.304714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 653.314597] alg: No test for adler32 (adler32-zlib) [ 654.270115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.332625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 654.357060] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 656.692809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 656.715969] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 659.249564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 659.275896] LNet: Removed LNI 10.1.2.3@tcp [ 659.295425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 659.340285] LNet: Removed LNI 10.1.2.31@tcp [ 659.362235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 659.396989] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 660.362254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 660.394658] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 661.382115] LNet: 39558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.390769] LNetError: 39558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 661.408138] LNet: Removed LNI 10.1.2.31@tcp [ 661.428495] LNet: Removed LNI 10.1.2.3@tcp [ 662.091615] Key type .llcrypt unregistered [ 662.093887] Key type ._llcrypt unregistered [ 674.724689] Key type ._llcrypt registered [ 674.728288] Key type .llcrypt registered [ 676.337865] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:51:45 (1786827105) [ 676.797352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 676.821855] alg: No test for adler32 (adler32-zlib) [ 677.757139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.812446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 677.839750] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 678.729792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.753676] LNet: 40517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.768460] LNetError: 40517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.780261] LNet: Removed LNI 192.168.206.16@tcp [ 678.812716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.884929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.16@tcp [ 678.919909] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 679.800449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.16@tcp [ 679.828903] LNet: Removed LNI 192.168.206.16@tcp [ 679.887816] LNet: 40652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.635476] Key type .llcrypt unregistered [ 680.638231] Key type ._llcrypt unregistered [ 691.158158] Key type ._llcrypt registered [ 691.160886] Key type .llcrypt registered [ 692.291250] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:52:02 (1786827122) [ 694.390332] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg616-server [ 696.282692] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:52:06 (1786827126) [ 696.660661] Key type .llcrypt unregistered [ 696.664117] Key type ._llcrypt unregistered [ 702.374282] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 702.624079] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 702.650529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 702.658673] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 705.166402] Key type ._llcrypt registered [ 705.168750] Key type .llcrypt registered [ 705.359834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 705.370485] alg: No test for adler32 (adler32-zlib) [ 706.328842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.413561] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 707.085416] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 709.152152] LNet: 42498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.158337] LNetError: 42498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.165252] LNet: Removed LNI 192.168.206.16@tcp1 [ 709.179272] LNet: Removed LNI 10.1.2.3@tcp2 [ 709.705857] Key type .llcrypt unregistered [ 709.709810] Key type ._llcrypt unregistered [ 719.274395] Key type ._llcrypt registered [ 719.276072] Key type .llcrypt registered [ 720.135309] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:52:30 (1786827150) [ 720.424061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 720.439677] alg: No test for adler32 (adler32-zlib) [ 721.311890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.347408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 721.373971] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 721.997551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.018428] LNet: 43472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.027334] LNetError: 43472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.048835] LNet: Removed LNI 192.168.206.16@tcp [ 722.071298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.121677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 722.141462] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 727.566586] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:52:37 (1786827157) [ 727.842472] LNet: 44063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.848785] LNetError: 44063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.868227] LNet: Removed LNI 192.168.206.16@tcp [ 728.319392] Key type .llcrypt unregistered [ 728.323246] Key type ._llcrypt unregistered [ 732.864501] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 732.939179] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 732.942743] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 732.950588] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 735.424663] Key type ._llcrypt registered [ 735.431285] Key type .llcrypt registered [ 735.585643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 735.600564] alg: No test for adler32 (adler32-zlib) [ 736.478969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.520565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 736.545982] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 737.208404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 737.221619] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 738.922955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.958458] LNet: 44807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.964895] LNetError: 44807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 738.975392] LNet: Removed LNI 10.1.2.3@tcp [ 739.002594] LNet: Removed LNI 192.168.206.16@tcp [ 739.036156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.113235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 739.148602] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 739.154091] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 739.768381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.792379] LNet: 44834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.800371] LNetError: 44834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 739.818731] LNetError: 44834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 739.828098] LNet: Removed LNI 10.1.2.3@tcp [ 739.842118] LNet: Removed LNI 192.168.206.16@tcp [ 739.861038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.920658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 739.951267] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 739.956549] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 740.397221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 740.429098] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 741.414803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.438792] LNet: 44997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.448310] LNetError: 44997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 741.457680] LNetError: 44997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 741.466681] LNet: Removed LNI 10.1.2.3@tcp [ 741.471819] LNet: Removed LNI 192.168.206.16@tcp [ 741.484594] LNet: Removed LNI 192.168.206.16@tcp2 [ 741.508515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.593905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 741.620325] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 741.624449] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 741.628994] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 748.036156] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:52:58 (1786827178) [ 748.151170] LNet: 45588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.162764] LNetError: 45588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 748.167642] LNetError: 45588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 748.179179] LNet: Removed LNI 10.1.2.3@tcp [ 748.182565] LNet: Removed LNI 192.168.206.16@tcp [ 748.201157] LNet: Removed LNI 192.168.206.16@tcp2 [ 748.756793] Key type .llcrypt unregistered [ 748.759327] Key type ._llcrypt unregistered [ 754.049298] Key type ._llcrypt registered [ 754.050845] Key type .llcrypt registered [ 754.213856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.239472] alg: No test for adler32 (adler32-zlib) [ 755.131945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.180357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.197783] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 755.879477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.900114] LNet: 45992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.919151] LNetError: 45992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.939853] LNet: Removed LNI 192.168.206.16@tcp [ 755.963927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.040587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 756.066399] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 756.583089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.602399] LNet: 46022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.613688] LNetError: 46022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 756.631404] LNet: Removed LNI 192.168.206.16@tcp [ 756.655820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.718983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 756.743824] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 757.126408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.163337] LNet: Removed LNI 192.168.206.16@tcp [ 757.186451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.243510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 757.264650] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 757.565474] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 757.716321] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 757.724197] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 757.733353] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 760.142635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.165501] LNet: 46129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.170625] LNet: 46129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 760.176896] LNetError: 46129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.180864] LNetError: 46129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 760.197356] LNet: Removed LNI 192.168.206.16@tcp [ 760.220962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.299359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 760.339654] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 765.859759] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:53:16 (1786827196) [ 765.984271] LNet: 46723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.999287] LNetError: 46723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.020100] LNet: Removed LNI 192.168.206.16@tcp [ 766.548555] Key type .llcrypt unregistered [ 766.557942] Key type ._llcrypt unregistered [ 772.354801] Key type ._llcrypt registered [ 772.360788] Key type .llcrypt registered [ 772.525962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.536748] alg: No test for adler32 (adler32-zlib) [ 773.502069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.552871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.582149] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 774.252489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.271229] LNet: 47129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.275988] LNetError: 47129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.296342] LNet: Removed LNI 192.168.206.16@tcp [ 774.314958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.371969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 774.394965] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 780.004883] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:53:30 (1786827210) [ 780.094119] LNet: 47715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.102855] LNetError: 47715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.113071] LNet: Removed LNI 192.168.206.16@tcp [ 780.716261] Key type .llcrypt unregistered [ 780.719292] Key type ._llcrypt unregistered [ 785.608302] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 785.696632] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 785.701647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 785.704866] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 788.183373] Key type ._llcrypt registered [ 788.186590] Key type .llcrypt registered [ 788.400051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.422868] alg: No test for adler32 (adler32-zlib) [ 789.341305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.410345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.438063] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 790.204967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.224712] LNet: 48176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.229689] LNetError: 48176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.240326] LNet: Removed LNI 192.168.206.16@tcp [ 790.269619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.328643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 790.355575] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 798.247665] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:53:47 (1786827227) [ 798.369390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.451937] LNet: 48770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.465868] LNetError: 48770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.507811] LNet: Removed LNI 192.168.206.16@tcp [ 798.558690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.725618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 804.878676] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:53:54 (1786827234) [ 805.033719] LNet: 49352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.853966] Key type .llcrypt unregistered [ 805.859458] Key type ._llcrypt unregistered [ 811.896449] Key type ._llcrypt registered [ 811.899197] Key type .llcrypt registered [ 812.173387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 812.193367] alg: No test for adler32 (adler32-zlib) [ 813.130665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.197365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 813.223174] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 814.266196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.292384] LNet: 49759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.302685] LNetError: 49759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.319811] LNet: Removed LNI 192.168.206.16@tcp [ 814.337666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.388818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 814.426844] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 821.643092] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:54:11 (1786827251) [ 821.737663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.760593] LNet: 50345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.767429] LNetError: 50345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.782971] LNet: Removed LNI 192.168.206.16@tcp [ 821.805811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.902827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 826.944778] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:54:16 (1786827256) [ 828.319214] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 829.779632] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:54:19 (1786827259) [ 831.233956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 833.030682] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:54:22 (1786827262) [ 834.557656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 836.089363] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:54:26 (1786827266) [ 837.538450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 839.014777] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:54:29 (1786827269) [ 840.538840] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 842.147597] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:54:32 (1786827272) [ 842.297832] LNet: 52399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.828279] Key type .llcrypt unregistered [ 842.832750] Key type ._llcrypt unregistered [ 848.460464] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 848.610921] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.621698] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 848.642533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 851.154849] Key type ._llcrypt registered [ 851.160176] Key type .llcrypt registered [ 851.338424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 851.361128] alg: No test for adler32 (adler32-zlib) [ 852.226928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.275230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 852.293297] LNet: Added LNI 192.168.206.16@tcp [8/512/0/180] [ 852.302338] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 852.305438] LNet: 52728:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 852.386123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.428492] LNet: 52747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.448299] LNetError: 52747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 852.456953] LNet: Removed LNI 10.1.2.3@tcp [ 852.459640] LNetError: 52747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.483358] LNet: Removed LNI 192.168.206.16@tcp [ 852.510494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.594596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 852.630462] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 852.698181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.723587] LNet: 52771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.731535] LNetError: 52771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.743530] LNet: Removed LNI 192.168.206.16@tcp [ 852.769360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.828849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 852.853809] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 852.858297] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 852.860154] LNet: 52779:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 852.952264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.984923] LNet: 52797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.995496] LNetError: 52797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.001874] LNet: Removed LNI 10.1.2.3@tcp [ 853.004514] LNetError: 52797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.024884] LNet: Removed LNI 192.168.206.16@tcp [ 853.044094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.113298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 853.134849] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 853.140430] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.143173] LNet: 52805:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 853.299719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.325393] LNet: 52831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.331270] LNetError: 52831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.342368] LNet: Removed LNI 10.1.2.3@tcp [ 853.348768] LNetError: 52831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.359821] LNet: Removed LNI 192.168.206.16@tcp [ 853.376608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.424443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 853.443050] LNet: Added LNI 192.168.206.16@tcp [8/256/0/182] [ 853.447528] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 853.543618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.567582] LNet: 52859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.576202] LNetError: 52859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.582540] LNet: Removed LNI 10.1.2.3@tcp [ 853.586057] LNetError: 52859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.600619] LNet: Removed LNI 192.168.206.16@tcp [ 853.627082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.702698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 853.725628] LNet: Added LNI 192.168.206.16@tcp [8/256/0/182] [ 853.730448] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 853.733273] LNet: 52867:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 853.814488] LNet: Removed LNI 10.1.2.3@tcp [ 853.830377] LNet: Removed LNI 192.168.206.16@tcp [ 854.347819] Key type .llcrypt unregistered [ 854.350055] Key type ._llcrypt unregistered [ 864.003634] Key type ._llcrypt registered [ 864.007823] Key type .llcrypt registered [ 865.072274] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 16:54:55 (1786827295) [ 866.347797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 868.066317] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:54:57 (1786827297) [ 868.362662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 868.375988] alg: No test for adler32 (adler32-zlib) [ 869.284398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.326684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 869.345321] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 869.957655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.16 [ 869.974665] LNet: 54134:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 869.984421] LNet: Removed LNI 192.168.206.16@tcp [ 876.063564] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:55:05 (1786827305) [ 876.238946] LNet: 54699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.253532] LNetError: 54699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.265563] LNet: Removed LNI 192.168.206.16@tcp [ 876.988809] Key type .llcrypt unregistered [ 876.993284] Key type ._llcrypt unregistered [ 883.314221] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 883.547850] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 886.269442] Key type ._llcrypt registered [ 886.271925] Key type .llcrypt registered [ 886.416798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 886.434798] alg: No test for adler32 (adler32-zlib) [ 887.620893] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 892.929480] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:55:23 (1786827323) [ 893.030046] LNet: 55612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.036091] LNetError: 55612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 893.063269] LNet: Removed LNI 10.1.2.3@tcp [ 893.718930] Key type .llcrypt unregistered [ 893.725836] Key type ._llcrypt unregistered [ 899.774876] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 899.953767] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 899.956737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 899.959195] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 902.571559] Key type ._llcrypt registered [ 902.574488] Key type .llcrypt registered [ 902.708618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.744848] alg: No test for adler32 (adler32-zlib) [ 903.727544] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 910.104467] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:55:39 (1786827339) [ 911.714813] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 913.156371] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:55:43 (1786827343) [ 913.273897] LNet: 56803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.282426] LNetError: 56803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 913.315570] LNet: Removed LNI 10.1.2.3@tcp [ 913.974863] Key type .llcrypt unregistered [ 913.979813] Key type ._llcrypt unregistered [ 919.885954] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 920.102809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 920.105793] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 920.109916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 922.856899] Key type ._llcrypt registered [ 922.858920] Key type .llcrypt registered [ 923.099924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 923.123562] alg: No test for adler32 (adler32-zlib) [ 924.013588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.072392] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 932.331479] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:56:02 (1786827362) [ 932.763972] LNet: 58045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.768615] LNetError: 58045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 932.789152] LNet: Removed LNI 10.1.2.3@tcp [ 933.337650] Key type .llcrypt unregistered [ 933.340219] Key type ._llcrypt unregistered [ 938.919772] Key type ._llcrypt registered [ 938.930435] Key type .llcrypt registered [ 939.149197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 939.172761] alg: No test for adler32 (adler32-zlib) [ 940.133597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 940.151397] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 946.678234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 947.544564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_interrupt -r 1 [ 947.574484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_interrupt -r 1 [ 947.611659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 947.643402] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 947.840860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 947.903930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 947.952883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 947.986275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 948.016690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 948.040269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 948.080597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 948.131470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 948.151697] LNet: 58486:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 948.170753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 948.204356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 948.236471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 948.267407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 948.290565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 948.319587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 948.337196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 948.693712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_dropped -r 1 [ 948.721291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_dropped -r 1 [ 948.782177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 948.803752] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 948.976426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 948.999381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 949.020419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 949.039878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 949.068594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 949.109286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 949.161477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 949.219946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 949.253724] LNet: 58595:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 949.271936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 949.309667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 949.343431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 949.390683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 949.408351] LNet: 1 local NIs in recovery (showing 1): 192.168.206.16@tcp [ 949.424949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 949.455722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 949.495862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 949.959648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_aborted -r 1 [ 949.989558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_aborted -r 1 [ 950.017955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 950.034501] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 950.038410] LNet: Skipped 1 previous similar message [ 950.198837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 950.235837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 950.264166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 950.299804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 950.337988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 950.363686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 950.386548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 950.412736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 950.432474] LNet: 58704:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 950.453593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 950.478895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 950.498923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 950.525737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 950.554489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 950.577993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 950.607299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 951.088730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_no_route -r 1 [ 951.139895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_no_route -r 1 [ 951.180864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 951.468042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 951.500306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 951.523401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 951.549325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 951.570032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 951.594208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 951.645841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 951.675217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 951.722682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 951.772107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 951.803142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 951.850667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 951.911626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 951.976866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 952.031287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 952.576560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_timeout -r 1 [ 952.617938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_timeout -r 1 [ 952.656571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 952.669160] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 952.672939] LNet: Skipped 2 previous similar messages [ 952.910896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 952.976119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 953.032661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 953.075365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 953.110977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 953.143435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 953.181179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 953.206725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 953.221615] LNet: 58923:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 953.225703] LNet: 58923:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 953.240137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 953.265955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 953.292409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 953.316353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 953.340404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 953.373751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 953.405041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 953.838561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_error -r 1 [ 953.875794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_error -r 1 [ 953.917718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 954.116973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 954.153355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 954.183910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 954.204176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 954.227335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 954.246290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 954.271303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 954.295188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 954.316199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 954.337599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 954.363873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 954.393436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 954.419416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 954.450426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 954.476941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 954.672346] LNet: 59077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 954.679573] LNetError: 59077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.753770] LNet: Removed LNI 192.168.206.16@tcp [ 956.080793] Key type .llcrypt unregistered [ 956.082171] Key type ._llcrypt unregistered [ 977.101426] Key type ._llcrypt registered [ 977.104172] Key type .llcrypt registered [ 978.354655] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:56:48 (1786827408) [ 979.276466] Key type .llcrypt unregistered [ 979.283648] Key type ._llcrypt unregistered [ 984.528431] Key type ._llcrypt registered [ 984.532563] Key type .llcrypt registered [ 984.796580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.805584] alg: No test for adler32 (adler32-zlib) [ 985.717795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 985.750948] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 992.164278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 1004.135594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.184610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1004.261633] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1005.642194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_interrupt -r 1 [ 1005.684853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_interrupt -r 1 [ 1005.715983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_interrupt -r 1 [ 1005.753189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_interrupt -r 1 [ 1005.810054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_interrupt -r 1 [ 1005.871612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_interrupt -r 1 [ 1005.901702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_interrupt -r 1 [ 1005.943829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_interrupt -r 1 [ 1005.985935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1006.029773] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1006.047798] LNet: 1 local NIs in recovery (showing 1): 192.168.206.16@tcp [ 1006.270825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1006.296668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1006.330885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1006.359329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1006.385274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1006.410593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1006.458964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1006.493250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1006.522316] LNet: 60549:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1006.550086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1006.581560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1006.631338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1006.663274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1006.704572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1006.740793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1006.786153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1007.216861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_dropped -r 1 [ 1007.260901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_dropped -r 1 [ 1007.311111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_dropped -r 1 [ 1007.351971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_dropped -r 1 [ 1007.389421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_dropped -r 1 [ 1007.431938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_dropped -r 1 [ 1007.471736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_dropped -r 1 [ 1007.510846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_dropped -r 1 [ 1007.558985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1007.582419] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1007.588981] LNet: Skipped 2 previous similar messages [ 1007.810693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1007.837861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1007.869099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1007.908977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1007.954596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1007.984934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1008.043276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1008.087030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1008.112898] LNet: 60671:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1008.136386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1008.174520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1008.201231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1008.224590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1008.249220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1008.275186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1008.304794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1008.736894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_aborted -r 1 [ 1008.762927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_aborted -r 1 [ 1008.788301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_aborted -r 1 [ 1008.813978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_aborted -r 1 [ 1008.843802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_aborted -r 1 [ 1008.873446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_aborted -r 1 [ 1008.900886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_aborted -r 1 [ 1008.926588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_aborted -r 1 [ 1008.968536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1008.990750] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1009.000167] LNet: Skipped 2 previous similar messages [ 1009.251915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1009.276812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1009.319424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1009.356734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1009.385873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1009.447959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1009.492619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1009.545315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1009.585554] LNet: 60793:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1009.608841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1009.632917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1009.661348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1009.697735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1009.724709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1009.769744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1009.811929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1010.341371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_no_route -r 1 [ 1010.408538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_no_route -r 1 [ 1010.494437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_no_route -r 1 [ 1010.539860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_no_route -r 1 [ 1010.565362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_no_route -r 1 [ 1010.592234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_no_route -r 1 [ 1010.620730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_no_route -r 1 [ 1010.648903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_no_route -r 1 [ 1010.688860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1010.864839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1010.888979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1010.912730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1010.942549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1010.969536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1011.003186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1011.043312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1011.066296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1011.104359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1011.140330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1011.197138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1011.260341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1011.305394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1011.349852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1011.375601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1011.965215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_timeout -r 1 [ 1012.027233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_timeout -r 1 [ 1012.074343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_timeout -r 1 [ 1012.134240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_timeout -r 1 [ 1012.200818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_timeout -r 1 [ 1012.251415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_timeout -r 1 [ 1012.299797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_timeout -r 1 [ 1012.339487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_timeout -r 1 [ 1012.409562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1012.443563] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1012.449719] LNet: Skipped 5 previous similar messages [ 1012.778749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1012.832589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1012.874525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1012.901059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1012.928180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1012.952561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1012.978314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1013.004710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1013.024639] LNet: 61035:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1013.032979] LNet: 61035:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1013.054909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1013.089858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1013.122808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1013.161846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1013.187563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1013.243649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1013.314800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1013.734958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_error -r 1 [ 1013.760974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_error -r 1 [ 1013.789962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_error -r 1 [ 1013.837177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_error -r 1 [ 1013.895226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_error -r 1 [ 1013.927779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_error -r 1 [ 1013.959277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_error -r 1 [ 1014.011173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_error -r 1 [ 1014.061524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1014.283642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1014.312260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1014.341393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1014.369800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1014.399950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1014.432689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1014.476145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1014.506600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1014.534712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1014.561648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1014.592539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1014.615861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1014.639599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1014.670219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1014.706859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1019.853940] LNet: 61237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.863836] LNetError: 61237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.873163] LNet: Removed LNI 192.168.206.16@tcp [ 1019.893675] LNet: Removed LNI 192.168.206.16@tcp1 [ 1020.266065] Key type .llcrypt unregistered [ 1020.270490] Key type ._llcrypt unregistered [ 1036.969077] Key type ._llcrypt registered [ 1036.970825] Key type .llcrypt registered [ 1038.065193] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:57:48 (1786827468) [ 1038.789732] Key type .llcrypt unregistered [ 1038.792843] Key type ._llcrypt unregistered [ 1044.431562] Key type ._llcrypt registered [ 1044.433890] Key type .llcrypt registered [ 1044.686829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1044.709342] alg: No test for adler32 (adler32-zlib) [ 1045.599639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1045.625945] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1052.765339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 1053.571115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e remote_dropped -r 1 [ 1053.603361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e remote_dropped -r 1 [ 1053.633371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1053.650694] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1053.868980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.893910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1053.919651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.943809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.965509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.988876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.017405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.040917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.056518] LNet: 62492:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1054.069412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.098989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.127692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1054.152697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1054.179930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1054.206690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1054.232597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1054.838681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e remote_error -r 1 [ 1054.875218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e remote_error -r 1 [ 1054.927552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1054.956659] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1055.130591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.158334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1055.181553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.203399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.226351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.257401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.281491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.310028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.331338] LNet: 62602:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1055.347564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1055.374988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1055.405357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1055.436389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1055.484518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1055.538128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1055.569901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1056.029505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e remote_timeout -r 1 [ 1056.057893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e remote_timeout -r 1 [ 1056.083824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1056.101899] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1056.274344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1056.298804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1056.347696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1056.400331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1056.443622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1056.478758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1056.527223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1056.565919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1056.584959] LNet: 62711:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1056.613167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1056.645381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.700525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.741671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.825789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.907637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.973293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1057.205920] LNet: 62756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.210925] LNetError: 62756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.227240] LNet: Removed LNI 192.168.206.16@tcp [ 1057.544573] Key type .llcrypt unregistered [ 1057.547274] Key type ._llcrypt unregistered [ 1072.670472] Key type ._llcrypt registered [ 1072.673418] Key type .llcrypt registered [ 1074.057594] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:58:23 (1786827503) [ 1075.001434] Key type .llcrypt unregistered [ 1075.005457] Key type ._llcrypt unregistered [ 1082.977339] Key type ._llcrypt registered [ 1082.979609] Key type .llcrypt registered [ 1083.144449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1083.174433] alg: No test for adler32 (adler32-zlib) [ 1084.255419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1084.324262] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1093.882156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 1102.663510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1102.744222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1102.768144] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1104.036646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e remote_dropped -r 1 [ 1104.064887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e remote_dropped -r 1 [ 1104.098472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e remote_dropped -r 1 [ 1104.128979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e remote_dropped -r 1 [ 1104.172576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e remote_dropped -r 1 [ 1104.235911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e remote_dropped -r 1 [ 1104.291562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e remote_dropped -r 1 [ 1104.348901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e remote_dropped -r 1 [ 1104.380479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1104.416259] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1104.422935] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.116@tcp [ 1104.680720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.709275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.734564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.756915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.781906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.804748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.832286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.891206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.918353] LNet: 64230:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.938838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.991582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.023435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.078611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.099424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.131187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.158488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.196567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.806925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e remote_error -r 1 [ 1105.835387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e remote_error -r 1 [ 1105.866425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e remote_error -r 1 [ 1105.896693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e remote_error -r 1 [ 1105.942178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e remote_error -r 1 [ 1106.001529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e remote_error -r 1 [ 1106.062741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e remote_error -r 1 [ 1106.125578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e remote_error -r 1 [ 1106.161684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1106.180600] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1106.186576] LNet: Skipped 2 previous similar messages [ 1106.434674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.475351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.516974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.561775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.610372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.651516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.706194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.763772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.796802] LNet: 64354:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1106.818418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.850548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.877104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.930290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.958746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.987222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.039576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1107.105918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.895771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e remote_timeout -r 1 [ 1107.936484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e remote_timeout -r 1 [ 1107.967538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e remote_timeout -r 1 [ 1107.993590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e remote_timeout -r 1 [ 1108.037222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e remote_timeout -r 1 [ 1108.082453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e remote_timeout -r 1 [ 1108.116270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e remote_timeout -r 1 [ 1108.151251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e remote_timeout -r 1 [ 1108.190475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.116@tcp [ 1108.211823] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1108.535695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.580529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.638954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.693981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.730765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.756129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.817871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.890902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.938878] LNet: 64477:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.963512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.016798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.074839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.124385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.177457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.226279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.262909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1109.307061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.727093] LNet: 64559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.733143] LNetError: 64559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.740695] LNet: Removed LNI 192.168.206.16@tcp [ 1113.767323] LNet: Removed LNI 192.168.206.16@tcp1 [ 1114.322954] Key type .llcrypt unregistered [ 1114.329787] Key type ._llcrypt unregistered [ 1130.921507] Key type ._llcrypt registered [ 1130.924072] Key type .llcrypt registered [ 1132.516697] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:59:21 (1786827561) [ 1133.813192] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1135.794023] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:59:25 (1786827565) [ 1136.488312] Key type .llcrypt unregistered [ 1136.491580] Key type ._llcrypt unregistered [ 1144.058360] Key type ._llcrypt registered [ 1144.064972] Key type .llcrypt registered [ 1144.273440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1144.310113] alg: No test for adler32 (adler32-zlib) [ 1145.324628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1145.386540] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1153.816972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 1154.357641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e network_timeout -r 1 [ 1154.426238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e network_timeout -r 1 [ 1154.500873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 1154.529858] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1154.598570] LNet: 1 local NIs in recovery (showing 1): 192.168.206.16@tcp [ 1154.762331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.791824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.824429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.862987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.907611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.966412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.998984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.038464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.057798] LNet: 66107:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.080512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.131953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.181274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.233647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.284892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.318980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.687057] LNet: 66150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.695747] LNetError: 66150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.728432] LNet: Removed LNI 192.168.206.16@tcp [ 1156.088460] Key type .llcrypt unregistered [ 1156.092305] Key type ._llcrypt unregistered [ 1174.408225] Key type ._llcrypt registered [ 1174.411312] Key type .llcrypt registered [ 1174.708553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1174.741348] alg: No test for adler32 (adler32-zlib) [ 1175.721866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1175.787152] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1182.582784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 1189.935827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1189.976335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1189.997787] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1191.422911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e network_timeout -r 1 [ 1191.461267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e network_timeout -r 1 [ 1191.489781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e network_timeout -r 1 [ 1191.544314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e network_timeout -r 1 [ 1191.581189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e network_timeout -r 1 [ 1191.614960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e network_timeout -r 1 [ 1191.666434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e network_timeout -r 1 [ 1191.703655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e network_timeout -r 1 [ 1191.737232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 1191.749340] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1191.892348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.982206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1192.019402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.049165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.084107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.108182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.129508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.160404] LNet: 1 local NIs in recovery (showing 1): 192.168.206.16@tcp [ 1192.162911] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.116@tcp [ 1192.186887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.223333] LNet: 67034:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.246234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.272176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.294420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.317665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.338368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.356966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.490556] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1197.505388] LNet: Skipped 2 previous similar messages [ 1198.018618] LNet: 67113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.024063] LNetError: 67113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.038981] LNet: Removed LNI 192.168.206.16@tcp [ 1198.052125] LNet: Removed LNI 192.168.206.16@tcp1 [ 1198.569114] Key type .llcrypt unregistered [ 1198.572396] Key type ._llcrypt unregistered [ 1215.213913] Key type ._llcrypt registered [ 1215.216233] Key type .llcrypt registered [ 1216.460370] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 17:00:46 (1786827646) [ 1217.011766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1217.036291] alg: No test for adler32 (adler32-zlib) [ 1218.062600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.132784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1218.168867] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1219.079655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1219.113853] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1219.997148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1220.050790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1220.095673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1220.123241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1220.148559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1220.171407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1220.195160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.16@tcp [ 1220.210112] LNetError: 68213:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.16@tcp to recovery [ 1221.216295] LNet: 1 local NIs in recovery (showing 1): 192.168.206.16@tcp [ 1221.216923] LNet: There was an unexpected network error while writing to 192.168.206.16: rc = -22 [ 1223.265049] LNet: There was an unexpected network error while writing to 192.168.206.16: rc = -22 [ 1223.666482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1227.363354] LNet: There was an unexpected network error while writing to 192.168.206.16: rc = -22 [ 1228.255362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1228.298545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1228.372987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1228.398509] LNet: 68331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.416421] LNetError: 68331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.427238] LNet: Removed LNI 192.168.206.16@tcp [ 1228.448382] LNet: Removed LNI 192.168.206.16@tcp1 [ 1228.466912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.522875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1228.549372] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1229.352925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1229.375684] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1230.662800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1230.746646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.786526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1230.852734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1230.947344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1231.042595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1231.124137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1231.162915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.16@tcp [ 1231.180464] LNetError: 68616:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.16@tcp to recovery [ 1231.586755] LNet: There was an unexpected network error while writing to 192.168.206.16: rc = -22 [ 1234.596844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1237.728638] LNet: There was an unexpected network error while writing to 192.168.206.16: rc = -22 [ 1237.752776] LNet: Skipped 1 previous similar message [ 1242.635417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1242.663663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1242.684360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.095794] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 17:01:18 (1786827678) [ 1249.215030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1249.255446] LNet: 69328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.268277] LNetError: 69328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.281300] LNetError: 69328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1249.296530] LNet: Removed LNI 192.168.206.16@tcp [ 1249.315317] LNet: Removed LNI 192.168.206.16@tcp1 [ 1249.358217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.428768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1249.464611] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1250.234616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1250.268826] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1250.974140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1251.052278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1251.098650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1251.149813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1251.181767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1251.219743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1251.253093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.16@tcp --health 0 [ 1251.349718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1251.431761] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.16@tcp [ 1252.449321] LNet: There was an unexpected network error while writing to 192.168.206.16: rc = -22 [ 1252.457474] LNet: Skipped 1 previous similar message [ 1266.859482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1267.011547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1267.039878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1267.066662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.16@tcp --health 500 [ 1267.176647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1274.991643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1275.063300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1275.107501] LNet: 69834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.128836] LNetError: 69834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.138633] LNetError: 69834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1275.145099] LNet: Removed LNI 192.168.206.16@tcp [ 1275.158637] LNet: Removed LNI 192.168.206.16@tcp1 [ 1275.185363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.250183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1275.272485] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1275.825386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1275.848559] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1276.605683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1276.638396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.695640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1276.718986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1276.742667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1276.779675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1276.803803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1276.837496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.16@tcp --health 0 [ 1277.284195] LNet: There was an unexpected network error while writing to 192.168.206.16: rc = -22 [ 1277.291826] LNet: Skipped 3 previous similar messages [ 1278.143687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1288.216321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1288.239764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1288.265982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1294.254777] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:02:04 (1786827724) [ 1294.692581] LNet: 70879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.698837] LNetError: 70879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.710159] LNetError: 70879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1294.721576] LNet: Removed LNI 192.168.206.16@tcp [ 1294.731993] LNet: Removed LNI 192.168.206.16@tcp1 [ 1295.280166] Key type .llcrypt unregistered [ 1295.282373] Key type ._llcrypt unregistered [ 1300.147156] Key type ._llcrypt registered [ 1300.149616] Key type .llcrypt registered [ 1300.340533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1300.378059] alg: No test for adler32 (adler32-zlib) [ 1301.372203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1301.401963] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1309.967162] Lustre: DEBUG MARKER: Initial discovery [ 1310.017985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.116@tcp [ 1311.992525] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1312.046638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1312.088375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.116@tcp [ 1312.112520] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1312.132257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1312.168440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.116@tcp -r 1 -m GET -l 3 [ 1312.204662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.116@tcp -r 1 -m GET -e local_error [ 1312.235190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.116@tcp -r 1 -m PUT -l 6 [ 1312.678178] LNet: 1 local NIs in recovery (showing 1): 192.168.206.16@tcp [ 1313.448566] Lustre: DEBUG MARKER: Force oleg616-client.virtnet to discover 192.168.206.116@tcp (in background) [ 1313.567880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.116@tcp [ 1314.639611] Lustre: DEBUG MARKER: Force oleg616-server to discover 192.168.206.16@tcp [ 1316.445903] Lustre: DEBUG MARKER: Wait for 71644 [ 1316.643375] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1317.895686] Lustre: DEBUG MARKER: Finished wait on 71644 [ 1318.100222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1318.129571] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1318.149658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1318.446754] LNet: 72099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.452507] LNetError: 72099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.526951] LNet: Removed LNI 192.168.206.16@tcp [ 1319.924610] Key type .llcrypt unregistered [ 1319.926443] Key type ._llcrypt unregistered [ 1334.965617] Key type ._llcrypt registered [ 1334.967762] Key type .llcrypt registered [ 1336.148791] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:02:46 (1786827766) [ 1336.685443] Key type .llcrypt unregistered [ 1336.690566] Key type ._llcrypt unregistered [ 1341.786274] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1341.929263] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1341.936210] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1341.953324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1344.473423] Key type ._llcrypt registered [ 1344.475856] Key type .llcrypt registered [ 1344.611625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1344.627755] alg: No test for adler32 (adler32-zlib) [ 1345.504156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.576541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1345.609747] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1346.204539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1346.254374] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1353.397283] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:03:03 (1786827783) [ 1353.494785] LNet: 74170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.503409] LNetError: 74170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1353.510357] LNet: Removed LNI 10.1.2.3@tcp [ 1353.524050] LNet: Removed LNI 192.168.206.16@tcp [ 1354.136330] Key type .llcrypt unregistered [ 1354.138032] Key type ._llcrypt unregistered [ 1359.729609] Key type ._llcrypt registered [ 1359.733318] Key type .llcrypt registered [ 1359.912576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1359.928867] alg: No test for adler32 (adler32-zlib) [ 1360.834134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.888550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1360.907228] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1361.448733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1361.464975] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1362.316415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.16@tcp [ 1363.360322] LNetError: Unexpected error -22 connecting to 192.168.206.16@tcp at host 192.168.206.16:988 [ 1367.439169] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:03:17 (1786827797) [ 1367.781026] LNet: 75312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.787060] LNetError: 75312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1367.797610] LNet: Removed LNI 10.1.2.3@tcp [ 1367.813966] LNet: Removed LNI 192.168.206.16@tcp [ 1368.356696] Key type .llcrypt unregistered [ 1368.361694] Key type ._llcrypt unregistered [ 1374.099860] Key type ._llcrypt registered [ 1374.103230] Key type .llcrypt registered [ 1374.248495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1374.258193] alg: No test for adler32 (adler32-zlib) [ 1375.174055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1375.214566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.235049] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1375.840570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1375.869437] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 1376.589244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.16@tcp1 --nid 192.168.206.16@tcp2 [ 1380.232404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.16@tcp1 192.168.206.16@tcp2 [ 1380.253169] LNetError: 76151:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.16@tcp1 to 192.168.206.16@tcp2 - Route table may be misconfigured [ 1380.285585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.16@tcp2 192.168.206.16@tcp1 [ 1385.216313] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:03:35 (1786827815) [ 1385.313851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1385.345881] LNet: 76720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1385.351948] LNetError: 76720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1385.361454] LNet: Removed LNI 192.168.206.16@tcp1 [ 1386.412654] LNet: Removed LNI 192.168.206.16@tcp2 [ 1386.432861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.476662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1386.495732] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1387.094796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1387.113377] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1387.871688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1387.907381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.16@tcp 192.168.206.16@tcp [ 1387.945605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.16@tcp1 192.168.206.16@tcp1 [ 1387.982776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.16@tcp -d 192.168.206.16@tcp -e network_timeout [ 1388.014704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -e network_timeout [ 1388.042868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -e network_timeout [ 1388.073866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -e network_timeout [ 1388.102886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1388.120558] LNet: There was an unexpected network error while writing to 192.168.206.16: rc = -22 [ 1388.145505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1388.238088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.326420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1388.513592] LNet: 1 local NIs in recovery (showing 1): 192.168.206.16@tcp [ 1393.174958] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:03:43 (1786827823) [ 1393.287839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1393.317715] LNet: 77588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.336331] LNetError: 77588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.341979] LNetError: 77588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1393.359079] LNet: Removed LNI 192.168.206.16@tcp [ 1394.410597] LNet: Removed LNI 192.168.206.16@tcp1 [ 1394.437366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.509520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1394.807965] LNet: 77629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.135163] Key type .llcrypt unregistered [ 1395.137751] Key type ._llcrypt unregistered [ 1405.252374] Key type ._llcrypt registered [ 1405.255203] Key type .llcrypt registered [ 1406.342096] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:03:56 (1786827836) [ 1406.637762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1406.652151] alg: No test for adler32 (adler32-zlib) [ 1407.562302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.764130] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1410.190303] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:04:00 (1786827840) [ 1410.284033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1410.327816] LNet: 78731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.363106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.419283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1410.454347] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1411.166727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1411.239089] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1412.091567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1412.132225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp1 [ 1412.190871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp1 [ 1417.632995] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:04:07 (1786827847) [ 1419.180331] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg616-server [ 1421.035133] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:04:10 (1786827850) [ 1423.336835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg616-server [ 1424.977382] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:04:14 (1786827854) [ 1426.892381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg616-server [ 1428.671151] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:04:18 (1786827858) [ 1430.205834] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg616-server [ 1431.895639] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:04:21 (1786827861) [ 1433.868539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg616-server [ 1435.596909] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:04:25 (1786827865) [ 1437.468536] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg616-server [ 1439.118452] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:04:29 (1786827869) [ 1441.064270] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg616-server [ 1442.740203] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:04:32 (1786827872) [ 1444.523329] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg616-server [ 1446.278870] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:04:36 (1786827876) [ 1448.513858] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg616-server [ 1450.197373] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:04:40 (1786827880) [ 1450.392409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.416917] LNet: 82613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.425474] LNetError: 82613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.432347] LNet: Removed LNI 192.168.206.16@tcp [ 1451.501302] LNet: Removed LNI 192.168.206.16@tcp1 [ 1451.522363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.609632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1451.641060] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1452.497287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1452.620187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1452.799059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1452.829195] LNet: 82760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.848405] LNetError: 82760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.862388] LNetError: 82760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1452.907214] LNet: Removed LNI 192.168.206.16@tcp [ 1452.940657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.069926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1453.097276] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1453.900636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1454.153966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.187967] LNetError: 82905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.210340] LNet: Removed LNI 192.168.206.16@tcp [ 1454.240180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.298285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1454.341378] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1454.976302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1455.041961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1456.353728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1456.377605] LNet: 83062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.382532] LNet: 83062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1456.390318] LNetError: 83062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.407158] LNet: Removed LNI 192.168.206.16@tcp [ 1456.421816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1456.483735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1456.502853] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1457.214814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1457.444762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.535048] LNet: Removed LNI 192.168.206.16@tcp [ 1458.563663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.618639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1458.654184] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1459.431896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1459.501943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1460.938499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.985788] LNet: 83363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.993994] LNet: 83363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1461.003519] LNetError: 83363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.009801] LNetError: 83363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1461.031647] LNet: Removed LNI 192.168.206.16@tcp [ 1461.068518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.142327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1461.172764] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1462.062642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1462.290326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.336721] LNet: Removed LNI 192.168.206.16@tcp [ 1463.382436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.436973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1463.467737] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1464.213452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1464.295696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1464.626466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1465.707687] LNet: Removed LNI 192.168.206.16@tcp [ 1465.760169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.838825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1465.864803] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1466.632438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1466.949677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1468.008316] LNet: Removed LNI 192.168.206.16@tcp [ 1468.064307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.142149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1468.173531] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1469.074271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1469.188871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1469.482331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.517371] LNet: 83948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.522658] LNet: 83948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1469.534549] LNetError: 83948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.541405] LNetError: 83948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1469.549852] LNet: Removed LNI 192.168.206.16@tcp [ 1469.566617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.620480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1469.636870] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1470.165780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1470.619418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.719908] LNet: Removed LNI 192.168.206.16@tcp [ 1471.776253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.880695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.919081] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1472.665272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1472.744870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1473.032563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1474.093963] LNet: Removed LNI 192.168.206.16@tcp [ 1474.135691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.218757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1474.257524] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1475.109292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1475.394705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1476.455244] LNet: Removed LNI 192.168.206.16@tcp [ 1476.496651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.614618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1476.644487] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1477.418769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1477.485229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1477.528820] LNet: Removed LNI 192.168.206.16@tcp [ 1477.547086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.590859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1477.618176] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1478.293647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1478.398875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.442331] LNet: Removed LNI 192.168.206.16@tcp [ 1478.455971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.503654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1478.560800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.596856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.662867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1478.744692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.769245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.817636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1478.866281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.899555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.954423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1478.994056] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1479.684325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.719472] LNet: Removed LNI 192.168.206.16@tcp [ 1479.735848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.811660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1485.891628] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:05:15 (1786827915) [ 1486.097580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1486.116985] LNet: 85407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1486.124674] LNet: 85407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1486.149435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.189635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1486.207501] LNet: Added LNI 192.168.206.16@tcp231 [8/256/0/180] [ 1487.272973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1487.296294] LNetError: 85545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1487.305166] LNetError: 85545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1487.321928] LNet: Removed LNI 192.168.206.16@tcp231 [ 1487.343558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1487.460140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1487.480764] LNet: Added LNI 192.168.206.16@tcp231 [8/256/0/180] [ 1487.492289] LNet: 85553:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1487.946342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1487.980374] LNet: Removed LNI 192.168.206.16@tcp231 [ 1487.995326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1488.014505] LNet: Added LNI 192.168.206.16@tcp231 [8/256/0/180] [ 1488.872536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1488.900276] LNet: Removed LNI 192.168.206.16@tcp231 [ 1488.912896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.987630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1489.019224] LNet: Added LNI 192.168.206.16@tcp231 [8/256/0/0] [ 1489.024355] LNet: 85705:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1489.451713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1489.481346] LNet: Removed LNI 192.168.206.16@tcp231 [ 1489.498966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1489.535680] LNet: Added LNI 192.168.206.16@tcp231 [8/256/0/0] [ 1490.588657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1490.625879] LNet: Removed LNI 192.168.206.16@tcp231 [ 1490.647983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.711853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1490.731251] LNet: Added LNI 192.168.206.16@tcp231 [8/256/0/60] [ 1490.736313] LNet: 85857:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1491.167361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1491.199834] LNet: Removed LNI 192.168.206.16@tcp231 [ 1491.213284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1491.242047] LNet: Added LNI 192.168.206.16@tcp231 [8/256/0/60] [ 1492.622827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.677160] LNet: Removed LNI 192.168.206.16@tcp231 [ 1492.692834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.751884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1492.784740] LNet: Added LNI 192.168.206.16@tcp231 [8/256/0/180] [ 1492.792459] LNet: 86008:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1493.314134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1493.345128] LNet: Removed LNI 192.168.206.16@tcp231 [ 1493.375981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1493.407578] LNet: Added LNI 192.168.206.16@tcp231 [8/256/0/180] [ 1499.958779] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:05:30 (1786827930) [ 1500.167957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.191286] LNet: Removed LNI 192.168.206.16@tcp231 [ 1500.208684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.271406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1500.289869] LNet: Added LNI 192.168.206.16@tcp232 [8/256/0/180] [ 1500.809764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1500.837615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1500.866760] LNet: Removed LNI 192.168.206.16@tcp232 [ 1500.889867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1500.916108] LNet: Added LNI 192.168.206.16@tcp232 [8/256/0/180] [ 1501.663769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1501.711577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1501.741075] LNet: Removed LNI 192.168.206.16@tcp232 [ 1501.757844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1501.794176] LNet: Added LNI 192.168.206.16@tcp232 [8/256/0/180] [ 1502.442851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1502.492991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1502.513834] LNet: Removed LNI 192.168.206.16@tcp232 [ 1502.530280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1502.565888] LNet: Added LNI 192.168.206.16@tcp232 [8/256/0/180] [ 1503.224666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1503.299853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1503.333577] LNet: Removed LNI 192.168.206.16@tcp232 [ 1503.352781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1508.954634] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:05:38 (1786827938) [ 1509.861132] Key type .llcrypt unregistered [ 1509.866851] Key type ._llcrypt unregistered [ 1516.420359] Key type ._llcrypt registered [ 1516.428840] Key type .llcrypt registered [ 1516.646851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1516.663731] alg: No test for adler32 (adler32-zlib) [ 1517.587846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1517.623619] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1526.249932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 1534.906766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.010505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1535.057536] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1536.579979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_interrupt -r 2 [ 1536.606514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_interrupt -r 2 [ 1536.647523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_interrupt -r 2 [ 1536.701905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_interrupt -r 2 [ 1536.758064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_interrupt -r 2 [ 1536.795579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_interrupt -r 2 [ 1536.845950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_interrupt -r 2 [ 1536.903190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_interrupt -r 2 [ 1536.967830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.16@tcp 192.168.206.116@tcp [ 1536.995543] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1537.003851] LNet: 1 local NIs in recovery (showing 1): 192.168.206.16@tcp [ 1537.376393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1537.424377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1537.470727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1537.513398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1537.558914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1537.589319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1537.613953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1537.643359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1537.663412] LNet: 88514:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1537.680594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1537.708780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1537.737808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1537.767590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1537.796978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1537.839270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1538.509522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_dropped -r 2 [ 1538.551614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_dropped -r 2 [ 1538.595903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_dropped -r 2 [ 1538.631610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_dropped -r 2 [ 1538.670335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_dropped -r 2 [ 1538.695684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_dropped -r 2 [ 1538.733210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_dropped -r 2 [ 1538.768770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_dropped -r 2 [ 1538.794305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.16@tcp 192.168.206.116@tcp [ 1538.824008] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1539.065867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1539.099618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1539.149791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1539.197305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1539.251579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1539.280810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1539.312534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1539.336605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1539.356227] LNet: 88634:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1539.373095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1539.404508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1539.436073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1539.467404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1539.489968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1539.523404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1540.064379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_aborted -r 2 [ 1540.102392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_aborted -r 2 [ 1540.164730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_aborted -r 2 [ 1540.217174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_aborted -r 2 [ 1540.272361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_aborted -r 2 [ 1540.311738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_aborted -r 2 [ 1540.355044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_aborted -r 2 [ 1540.398336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_aborted -r 2 [ 1540.430973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.16@tcp 192.168.206.116@tcp [ 1540.456647] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1540.727618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1540.759541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1540.782244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1540.809289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1540.880804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1540.919978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1540.955333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1540.995920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1541.027672] LNet: 88754:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1541.055605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1541.086840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1541.123544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1541.158504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1541.202553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1541.242038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1541.664442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_no_route -r 2 [ 1541.697827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_no_route -r 2 [ 1541.725478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_no_route -r 2 [ 1541.752045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_no_route -r 2 [ 1541.792410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_no_route -r 2 [ 1541.826890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_no_route -r 2 [ 1541.859752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_no_route -r 2 [ 1541.910276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_no_route -r 2 [ 1541.951491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.16@tcp 192.168.206.116@tcp [ 1542.244569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1542.278027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1542.311617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1542.353741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1542.391687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1542.419050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1542.468921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1542.502348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1542.570277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1542.622377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1542.710159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1542.770316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1542.805729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1542.838947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1543.502111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e local_timeout -r 2 [ 1543.528623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e local_timeout -r 2 [ 1543.550489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e local_timeout -r 2 [ 1543.576505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e local_timeout -r 2 [ 1543.599576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e local_timeout -r 2 [ 1543.629882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e local_timeout -r 2 [ 1543.667269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e local_timeout -r 2 [ 1543.702529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e local_timeout -r 2 [ 1543.728721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.16@tcp 192.168.206.116@tcp [ 1543.754975] LNet: There was an unexpected network error while writing to 192.168.206.116: rc = -22 [ 1543.765615] LNet: Skipped 1 previous similar message [ 1544.135625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1544.175343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1544.231630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1544.296955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1544.373680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1544.422992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1544.461314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1544.494150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1544.511847] LNet: 88995:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1544.517961] LNet: 88995:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1544.541118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1544.569934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1544.605475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1544.632799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1544.672095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1544.723800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1545.640533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp -m GET -e remote_dropped -r 2 [ 1545.678898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.116@tcp1 -m GET -e remote_dropped -r 2 [ 1545.710145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp -m GET -e remote_dropped -r 2 [ 1545.743411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp -d 192.168.206.16@tcp1 -m GET -e remote_dropped -r 2 [ 1545.787121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp -m GET -e remote_dropped -r 2 [ 1545.836271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.16@tcp1 -d 192.168.206.116@tcp1 -m GET -e remote_dropped -r 2 [ 1545.891426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp -m GET -e remote_dropped -r 2 [ 1545.946390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.16@tcp1 -d 192.168.206.16@tcp1 -m GET -e remote_dropped -r 2 [ 1545.986791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.16@tcp 192.168.206.116@tcp [ 1546.016397] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.116@tcp [ 1546.325778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1546.367115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1546.408729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1546.460569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1546.504520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1546.552505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1546.591812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1546.633599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1546.668438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1546.716271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1546.744447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1546.780897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1546.837655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1546.876630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1552.488158] LNet: 89194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.494381] LNetError: 89194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.507339] LNet: Removed LNI 192.168.206.16@tcp [ 1552.541816] LNet: Removed LNI 192.168.206.16@tcp1 [ 1553.125836] Key type .llcrypt unregistered [ 1553.134872] Key type ._llcrypt unregistered [ 1576.032658] Key type ._llcrypt registered [ 1576.037152] Key type .llcrypt registered [ 1577.151916] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:06:47 (1786828007) [ 1577.795534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1577.847985] alg: No test for adler32 (adler32-zlib) [ 1578.789568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.846875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.867954] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1579.762251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1579.823223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.16@tcp [ 1579.836569] LNetError: 90174:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.16@tcp to recovery [ 1579.873252] LNet: 1 local NIs in recovery (showing 1): 192.168.206.16@tcp [ 1579.882073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.16@tcp [ 1580.002741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1586.522263] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:06:56 (1786828016) [ 1586.668488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.723827] LNet: 90758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.739555] LNetError: 90758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.761805] LNet: Removed LNI 192.168.206.16@tcp [ 1586.830245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.895845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1586.928338] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1587.910193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1587.933513] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1588.982784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.16@tcp [ 1589.011672] LNetError: 91030:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.16@tcp to recovery [ 1589.925318] LNet: There was an unexpected network error while writing to 192.168.206.16: rc = -22 [ 1590.324608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.16@tcp [ 1600.101928] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:07:08 (1786828028) [ 1601.784238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg616-server [ 1603.817459] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:07:13 (1786828033) [ 1606.747245] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg616-server [ 1609.187238] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:07:18 (1786828038) [ 1611.031421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg616-server [ 1612.792412] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:07:22 (1786828042) [ 1612.867675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.900105] LNet: 92651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1612.917346] LNetError: 92651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.984196] LNet: Removed LNI 192.168.206.16@tcp [ 1613.987579] LNetError: 92651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.006927] LNet: Removed LNI 192.168.206.16@tcp1 [ 1614.067336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.152770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.188513] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1614.954336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1615.022782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1615.063732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1615.166255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1615.199937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1615.238829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1620.773487] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:07:30 (1786828050) [ 1620.986976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.008461] LNet: 93386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.013982] LNetError: 93386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1621.031388] LNet: Removed LNI 192.168.206.16@tcp [ 1621.056733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.109562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1621.129767] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1628.291383] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:07:38 (1786828058) [ 1628.493678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.532655] LNet: 94089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1628.543781] LNetError: 94089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1628.555621] LNet: Removed LNI 192.168.206.16@tcp [ 1628.574066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.616777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1628.648576] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1635.693168] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:07:45 (1786828065) [ 1637.173427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1638.858656] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:07:48 (1786828068) [ 1639.415978] LNet: 95128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.424309] LNetError: 95128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.445742] LNet: Removed LNI 192.168.206.16@tcp [ 1640.021685] Key type .llcrypt unregistered [ 1640.023420] Key type ._llcrypt unregistered [ 1645.291300] Key type ._llcrypt registered [ 1645.295864] Key type .llcrypt registered [ 1645.464646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1645.475575] alg: No test for adler32 (adler32-zlib) [ 1646.360616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1646.393154] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1652.675992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 1656.654558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.116@tcp [ 1657.008322] LNet: 95552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.013669] LNetError: 95552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.027219] LNet: Removed LNI 192.168.206.16@tcp [ 1657.420144] Key type .llcrypt unregistered [ 1657.423422] Key type ._llcrypt unregistered [ 1668.368263] Key type ._llcrypt registered [ 1668.370541] Key type .llcrypt registered [ 1669.435644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1671.001285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1672.352925] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:08:22 (1786828102) [ 1673.526204] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1674.927550] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:08:25 (1786828105) [ 1676.524255] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg616-server [ 1678.128752] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:08:28 (1786828108) [ 1679.865919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg616-server [ 1681.919728] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:08:31 (1786828111) [ 1683.311340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1683.388521] alg: No test for adler32 (adler32-zlib) [ 1684.464838] Lustre: Lustre: Build Version: 2.17.57_4_gbf75d46 [ 1684.756376] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1686.449331] Key type lgssc registered [ 1687.561474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1704.823384] Key type lgssc unregistered [ 1705.064726] LNet: 98731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.073872] LNetError: 98731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.109594] LNet: Removed LNI 192.168.206.16@tcp [ 1705.805598] Key type .llcrypt unregistered [ 1705.808136] Key type ._llcrypt unregistered [ 1720.862197] Key type ._llcrypt registered [ 1720.867582] Key type .llcrypt registered [ 1722.233171] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:09:11 (1786828151) [ 1722.757703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1722.774880] alg: No test for adler32 (adler32-zlib) [ 1723.713160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.789523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1723.842590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1723.890756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1724.028461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1730.146451] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:09:19 (1786828159) [ 1730.344406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.361783] LNet: 100148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.381193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.420464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1730.441807] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1731.162670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.180322] LNet: 100292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.185328] LNetError: 100292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.194828] LNet: Removed LNI 192.168.206.16@tcp [ 1731.216186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.281703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1731.316536] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1731.783498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.798393] LNetError: 100320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.817447] LNet: Removed LNI 192.168.206.16@tcp [ 1731.840200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.895488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1731.942769] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1738.441416] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:09:28 (1786828168) [ 1738.540065] LNet: 100902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.551983] LNet: 100902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1738.559470] LNetError: 100902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.573573] LNet: Removed LNI 192.168.206.16@tcp [ 1739.639744] Key type .llcrypt unregistered [ 1739.644501] Key type ._llcrypt unregistered [ 1747.344510] Key type ._llcrypt registered [ 1747.348957] Key type .llcrypt registered [ 1747.549534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1747.567898] alg: No test for adler32 (adler32-zlib) [ 1748.533755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.601816] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1748.605781] LNet: *** cfs_fail_loc=e003, val=100*** [ 1749.447333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1749.487685] LNet: *** cfs_fail_loc=e003, val=100*** [ 1749.491391] LNet: Skipped 2 previous similar messages [ 1749.499486] Ping reply received with mapping: [ 1749.501313] found metadata reply NI[192.168.206.16@tcp] [ 1749.507443] 10 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1749.514822] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1749.516942] 5a 5a 5a 5a 5a 5a 5a 5a [ 1749.676898] LNet: 101312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.699883] LNetError: 101312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.732897] LNet: Removed LNI 192.168.206.16@tcp [ 1750.495501] Key type .llcrypt unregistered [ 1750.497955] Key type ._llcrypt unregistered [ 1762.529911] Key type ._llcrypt registered [ 1762.531492] Key type .llcrypt registered [ 1763.672161] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:09:53 (1786828193) [ 1764.329340] Key type .llcrypt unregistered [ 1764.335589] Key type ._llcrypt unregistered [ 1769.838472] Key type ._llcrypt registered [ 1769.841027] Key type .llcrypt registered [ 1770.111285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1770.133283] alg: No test for adler32 (adler32-zlib) [ 1771.448397] LNetError: 102369:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1771.920557] Key type .llcrypt unregistered [ 1771.923593] Key type ._llcrypt unregistered [ 1777.233460] Key type ._llcrypt registered [ 1777.234740] Key type .llcrypt registered [ 1778.265959] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:10:08 (1786828208) [ 1779.561427] Key type .llcrypt unregistered [ 1779.566856] Key type ._llcrypt unregistered [ 1787.109096] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1787.117202] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1789.954337] Key type ._llcrypt registered [ 1789.960298] Key type .llcrypt registered [ 1790.185956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1790.209199] alg: No test for adler32 (adler32-zlib) [ 1791.165181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.266484] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1801.172172] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:10:30 (1786828230) [ 1801.299422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.334254] LNet: 104154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.342889] LNetError: 104154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1801.371937] LNet: Removed LNI 10.1.2.3@tcp [ 1801.409100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1802.075447] Key type .llcrypt unregistered [ 1802.077291] Key type ._llcrypt unregistered [ 1808.642085] Key type ._llcrypt registered [ 1808.647619] Key type .llcrypt registered [ 1808.826337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1808.848730] alg: No test for adler32 (adler32-zlib) [ 1809.749452] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1809.823950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1809.966455] LNet: 104451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1809.972962] LNetError: 104451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1809.983661] LNet: Removed LNI 192.168.206.16@tcp [ 1810.587427] Key type .llcrypt unregistered [ 1810.598351] Key type ._llcrypt unregistered [ 1820.750978] Key type ._llcrypt registered [ 1820.753801] Key type .llcrypt registered [ 1821.904563] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:10:51 (1786828251) [ 1822.466435] Key type .llcrypt unregistered [ 1822.469579] Key type ._llcrypt unregistered [ 1827.776453] Key type ._llcrypt registered [ 1827.781590] Key type .llcrypt registered [ 1827.970972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1827.986888] alg: No test for adler32 (adler32-zlib) [ 1833.964975] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:11:04 (1786828264) [ 1834.040774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.082497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.126931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.153751] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1834.585174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.613264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1834.642355] LNet: Removed LNI 192.168.206.16@tcp [ 1834.865139] LNet: 106206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.142239] Key type .llcrypt unregistered [ 1835.148478] Key type ._llcrypt unregistered [ 1843.733596] Key type ._llcrypt registered [ 1843.741219] Key type .llcrypt registered [ 1844.824404] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:11:14 (1786828274) [ 1845.378929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1845.399953] alg: No test for adler32 (adler32-zlib) [ 1846.255382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.304173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1846.327934] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1846.909945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1846.974872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.16@tcp [ 1846.993548] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1847.000343] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1847.003789] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1846, 0, 1896 [ 1847.007177] LNet: Conn 0 sched [nconns, cpt]: 2, 0 [ 1847.010378] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1847.013536] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1847.018838] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1846, 0, 1896 [ 1847.021202] LNet: Conn 1 sched [nconns, cpt]: 2, 0 [ 1854.073718] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:11:23 (1786828283) [ 1854.513400] LNet: 107745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.518710] LNetError: 107745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.535177] LNet: Removed LNI 192.168.206.16@tcp [ 1855.333403] Key type .llcrypt unregistered [ 1855.339878] Key type ._llcrypt unregistered [ 1863.268484] Key type ._llcrypt registered [ 1863.274114] Key type .llcrypt registered [ 1863.676504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1863.694608] alg: No test for adler32 (adler32-zlib) [ 1864.726146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1864.764352] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1872.840316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 1881.673814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.735954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1881.775990] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 1882.838688] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1883.005980] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1883.008410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1883.013299] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1885.339870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1885.361571] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1895.608228] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:12:05 (1786828325) [ 1896.064801] LNet: 109258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.073545] LNetError: 109258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.090985] LNet: Removed LNI 192.168.206.16@tcp [ 1897.120171] LNet: Removed LNI 192.168.206.16@tcp1 [ 1897.131621] LNetError: 109258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1897.136687] LNetError: 109258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1897.145744] LNet: Removed LNI 10.1.2.3@tcp99 [ 1897.920659] Key type .llcrypt unregistered [ 1897.922997] Key type ._llcrypt unregistered [ 1903.880215] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1903.882475] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1906.799591] Key type ._llcrypt registered [ 1906.813564] Key type .llcrypt registered [ 1907.099465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1907.116159] alg: No test for adler32 (adler32-zlib) [ 1908.171681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.241366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1908.274438] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1909.000522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1909.028533] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1911.057642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1911.107436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1911.200429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1911.238079] LNetError: 110006:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1911.310485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1911.343883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1911.617878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1911.685161] LNet: 110026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.692735] LNetError: 110026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1911.700331] LNet: Removed LNI 10.1.2.3@tcp [ 1912.741640] LNet: Removed LNI 192.168.206.16@tcp [ 1913.537853] Key type .llcrypt unregistered [ 1913.541279] Key type ._llcrypt unregistered [ 1924.001548] Key type ._llcrypt registered [ 1924.003316] Key type .llcrypt registered [ 1925.080541] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:12:35 (1786828355) [ 1925.574408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.589302] alg: No test for adler32 (adler32-zlib) [ 1926.504792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.540370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1926.562842] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1927.110849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 1927.196638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg616-client@tcp [ 1933.170781] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:12:43 (1786828363) [ 1933.388820] LNet: 111561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.413699] LNetError: 111561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.431610] LNet: Removed LNI 192.168.206.16@tcp [ 1934.037695] Key type .llcrypt unregistered [ 1934.040127] Key type ._llcrypt unregistered [ 1939.795127] Key type ._llcrypt registered [ 1939.802676] Key type .llcrypt registered [ 1940.039204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1940.048747] alg: No test for adler32 (adler32-zlib) [ 1940.926996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1940.956764] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1941.002858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1946.353607] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:12:56 (1786828376) [ 1947.773928] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1949.526682] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:12:59 (1786828379) [ 1950.748678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1952.620986] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:13:02 (1786828382) [ 1952.682327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1952.701136] LNet: 112977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.712075] LNetError: 112977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.733234] LNet: Removed LNI 192.168.206.16@tcp [ 1952.757223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.854652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1952.889660] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1953.790158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1953.841901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1959.393759] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:13:09 (1786828389) [ 1959.465222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1959.478676] LNet: 113687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1959.483972] LNetError: 113687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1959.497102] LNet: Removed LNI 192.168.206.16@tcp [ 1959.518116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.605661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1959.638764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1960.156875] Key type .llcrypt unregistered [ 1960.159309] Key type ._llcrypt unregistered [ 1969.937788] Key type ._llcrypt registered [ 1969.943353] Key type .llcrypt registered [ 1971.039360] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:13:21 (1786828401) [ 1971.391433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1971.406381] alg: No test for adler32 (adler32-zlib) [ 1972.290659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1972.338871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1972.363528] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1972.924239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1972.975813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1978.332740] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:13:28 (1786828408) [ 1978.568993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.590432] LNet: 115252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.607494] LNetError: 115252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.624697] LNet: Removed LNI 192.168.206.16@tcp [ 1978.646738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.723102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1978.763298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1984.299469] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:13:34 (1786828414) [ 1984.473794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1984.512779] LNet: 115832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.562381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.662880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1985.136868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1985.199125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1985.257225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1985.768850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1985.804089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1985.894880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.917372] LNet: 115865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.946064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1986.023318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1986.480403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1986.508503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1986.543197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1986.845308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1986.896869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1992.929590] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:13:42 (1786828422) [ 1993.286137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1993.329283] LNet: 116465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.364766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.412937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1993.439971] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1994.095921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1994.120251] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 1994.847566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1994.898506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1995.078513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1995.093781] LNetError: 116744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1995.101618] LNet: Removed LNI 192.168.206.16@tcp [ 1995.120402] LNet: Removed LNI 192.168.206.16@tcp2 [ 1995.146093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1995.221959] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1995.227092] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 1995.236476] LNet: 116750:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1995.665126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1995.895498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1996.960159] LNet: Removed LNI 192.168.206.16@tcp [ 1996.963652] LNetError: 116787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1996.970036] LNetError: 116787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1996.988684] LNet: Removed LNI 192.168.206.16@tcp2 [ 1997.010075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1997.048256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1997.065204] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1997.759973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1997.789067] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 1998.474527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1998.512478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1998.568780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 1998.721532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1998.735771] LNet: 117072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.740458] LNet: 117072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1999.776359] LNet: Removed LNI 192.168.206.16@tcp [ 1999.778274] LNetError: 117072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1999.785636] LNetError: 117072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1999.800737] LNet: Removed LNI 192.168.206.16@tcp2 [ 1999.831801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1999.883496] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 1999.888619] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 1999.892512] LNet: 117078:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2000.233192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2000.436157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2000.470274] LNet: Removed LNI 192.168.206.16@tcp [ 2001.510686] LNet: Removed LNI 192.168.206.16@tcp2 [ 2001.547640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2001.603698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2001.620583] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2002.158826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2002.173581] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 2002.712761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2002.745944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2002.956756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.000277] LNet: Removed LNI 192.168.206.16@tcp [ 2004.003463] LNetError: 117395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2004.009493] LNetError: 117395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2004.026063] LNet: Removed LNI 192.168.206.16@tcp2 [ 2004.054473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.098748] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2004.102715] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 2004.109681] LNet: 117401:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2004.552269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2004.753850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.792175] LNet: Removed LNI 192.168.206.16@tcp [ 2005.804644] LNet: Removed LNI 192.168.206.16@tcp2 [ 2005.828693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.944772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2005.977098] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2006.728851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2006.743555] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 2007.470706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2007.499592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2007.545810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2007.747532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.766170] LNet: 117723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.771316] LNet: 117723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2008.800832] LNet: Removed LNI 192.168.206.16@tcp [ 2009.834649] LNet: Removed LNI 192.168.206.16@tcp2 [ 2009.863408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.956759] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2009.964421] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 2009.974906] LNet: 117729:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2010.466303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2010.684422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2010.705955] LNet: Removed LNI 192.168.206.16@tcp [ 2011.750795] LNet: Removed LNI 192.168.206.16@tcp2 [ 2011.788445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2011.854405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2011.882573] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2012.456802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2012.473792] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 2013.187629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2013.210601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2013.263950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2013.531900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2013.556702] LNetError: 118063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.576274] LNetError: 118063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2013.584091] LNet: Removed LNI 192.168.206.16@tcp [ 2013.602652] LNet: Removed LNI 192.168.206.16@tcp2 [ 2013.632596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.673922] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2013.680297] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 2014.041761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2014.354972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2015.392157] LNet: Removed LNI 192.168.206.16@tcp [ 2015.403188] LNet: Removed LNI 192.168.206.16@tcp2 [ 2015.437452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2015.513399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2015.541052] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2016.073194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2016.092795] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 2016.829319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2016.857800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2016.878454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2016.918859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2017.276068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2018.336165] LNet: Removed LNI 192.168.206.16@tcp [ 2019.368511] LNet: Removed LNI 192.168.206.16@tcp2 [ 2019.444393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2019.544655] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2019.555932] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 2019.564672] LNet: 118429:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2019.575475] LNet: 118429:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2020.217575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2027.022888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2028.647691] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:14:18 (1786828458) [ 2028.891911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2028.934875] LNet: 119189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.943829] LNet: 119189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2028.955836] LNet: Removed LNI 192.168.206.16@tcp [ 2030.000286] LNet: Removed LNI 192.168.206.16@tcp2 [ 2030.035966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.266135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2030.286692] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2030.994281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2031.006594] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2031.803624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.821918] LNetError: 119462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2031.827590] LNetError: 119462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2031.833712] LNet: Removed LNI 10.1.2.3@tcp [ 2031.840691] LNet: Removed LNI 192.168.206.16@tcp [ 2031.866937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2031.919695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2031.941843] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2031.945808] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2032.220741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2032.241140] LNet: Removed LNI 192.168.206.16@tcp [ 2032.251190] LNet: Removed LNI 10.1.2.3@tcp [ 2037.978826] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 17:14:28 (1786828468) [ 2038.244620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.282367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.323513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2038.342386] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2038.880547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2038.936122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 2038.993821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 2039.031938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 2039.067960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 2039.097869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 2039.132593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 2039.200790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 2039.258207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 2039.310072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 2039.360458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.16@tcp [ 2039.516439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2045.144417] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:14:35 (1786828475) [ 2045.299804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2046.374960] LNet: Removed LNI 192.168.206.16@tcp [ 2046.395512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.469540] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2046.619301] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2046.622227] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2046.625296] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2048.896548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2048.921750] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2049.479695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2049.515354] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2051.231889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2051.273570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2051.332265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.16@tcp -m PUT -e local_timeout -r 1 [ 2051.397366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2051.455426] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2051.457955] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2051.709173] LNetError: 120866:0:(lib-move.c:4801:lnet_parse()) 192.168.206.16@tcp, src 192.168.206.16@tcp: Dropping PUT (error -108 looking up sender) [ 2053.792167] LNet: Removed LNI 10.1.2.3@tcp [ 2054.827346] LNet: Removed LNI 192.168.206.16@tcp [ 2055.209048] Key type .llcrypt unregistered [ 2055.215829] Key type ._llcrypt unregistered [ 2065.401985] Key type ._llcrypt registered [ 2065.404629] Key type .llcrypt registered [ 2066.619796] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:14:56 (1786828496) [ 2067.218546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2067.241598] alg: No test for adler32 (adler32-zlib) [ 2068.174236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2068.503555] LNet: 122186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.115519] Key type .llcrypt unregistered [ 2069.118376] Key type ._llcrypt unregistered [ 2074.680953] Key type ._llcrypt registered [ 2074.683341] Key type .llcrypt registered [ 2074.811644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2074.825966] alg: No test for adler32 (adler32-zlib) [ 2075.749694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2075.816862] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2081.892902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 2082.127261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2082.915562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.116@tcp [ 2082.979184] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.116@tcp [ 2083.132459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2083.168612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2083.205357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2083.235258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2083.278430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2083.310291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2083.345759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2083.377053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2083.389867] LNet: 122639:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2083.417843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2083.457195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2083.505881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2083.535346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2083.586501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2083.620502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2084.185840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2084.443043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2084.479801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2084.505494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2084.536856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2084.591330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2084.625973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2084.653578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2084.677253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2084.691423] LNet: 122757:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2084.706123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2084.728355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2084.754518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2084.787593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2084.810967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2084.834653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2085.440056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.116@tcp [ 2085.686501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2085.729421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2085.765308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2085.810142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2085.840543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2085.876609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2085.924934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2085.974723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2085.995699] LNet: 122874:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2086.015291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2086.058043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2086.158562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2086.223773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2086.283424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2086.318759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2086.857449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2087.074761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2087.106597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2087.138958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2087.167995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2087.199209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2087.234953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2087.260709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2087.314907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2087.373791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2087.420988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2087.452890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2087.485389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2087.524455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2087.561587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2087.913926] LNet: 123041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.931994] LNetError: 123041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.954483] LNet: Removed LNI 192.168.206.16@tcp [ 2088.392636] Key type .llcrypt unregistered [ 2088.397289] Key type ._llcrypt unregistered [ 2103.265886] Key type ._llcrypt registered [ 2103.268439] Key type .llcrypt registered [ 2104.337212] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:15:34 (1786828534) [ 2104.600884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2104.615710] alg: No test for adler32 (adler32-zlib) [ 2105.519359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.975865] LNet: 123928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.563434] Key type .llcrypt unregistered [ 2106.574061] Key type ._llcrypt unregistered [ 2112.420452] Key type ._llcrypt registered [ 2112.423376] Key type .llcrypt registered [ 2112.713924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2112.746049] alg: No test for adler32 (adler32-zlib) [ 2113.671767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2113.691698] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2121.269836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.116@tcp [ 2128.998208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.059849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2129.089588] LNet: Added LNI 192.168.206.16@tcp1 [8/256/0/180] [ 2129.880765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2130.438331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.116@tcp [ 2130.626957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2130.660356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2130.682960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2130.700755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2130.721823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2130.743982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2130.763669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2130.782835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2130.793377] LNet: 124583:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2130.805627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2130.825904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2130.852343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2130.875180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2130.894375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2130.916259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2131.385896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2131.607841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2131.640840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2131.668760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2131.692322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2131.721370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2131.779201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2131.815037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2131.855153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2131.875558] LNet: 124701:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2131.888773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2131.923382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2131.950261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2131.986164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2132.035290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2132.061178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2132.626771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.116@tcp [ 2132.859961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2132.897854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2132.940577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2132.968728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2133.006572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2133.040344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2133.074224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2133.111727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2133.125466] LNet: 124818:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2133.140459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2133.152419] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.116@tcp [ 2133.174672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2133.213330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2133.245167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2133.271901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2133.294701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2133.807729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2134.012362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2134.047635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2134.081781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2134.117762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2134.156900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2134.189721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2134.253042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2134.301836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2134.342238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2134.379295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2134.427853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2134.475852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2134.509291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2134.539784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2139.184736] LNet: 125020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.189964] LNetError: 125020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.196781] LNet: Removed LNI 192.168.206.16@tcp [ 2139.207181] LNet: Removed LNI 192.168.206.16@tcp1 [ 2139.603126] Key type .llcrypt unregistered [ 2139.609022] Key type ._llcrypt unregistered [ 2153.459947] Key type ._llcrypt registered [ 2153.461884] Key type .llcrypt registered [ 2154.696491] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 17:16:24 (1786828584) [ 2154.979546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2154.996638] alg: No test for adler32 (adler32-zlib) [ 2155.883631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.930662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.948856] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2156.510900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2156.529152] LNet: Added LNI 192.168.206.16@tcp2 [8/256/0/180] [ 2157.152501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.16@tcp [ 2157.258333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2158.304233] LNet: Removed LNI 192.168.206.16@tcp [ 2163.866425] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 17:16:33 (1786828593) [ 2165.052644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2166.513686] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:16:36 (1786828596) [ 2173.188979] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:16:43 (1786828603) [ 2179.278947] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:16:49 (1786828609) [ 2185.948787] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:16:56 (1786828616) [ 2192.970306] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:17:02 (1786828622) [ 2199.711379] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:17:09 (1786828629) [ 2207.819987] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:17:17 (1786828637) [ 2214.952604] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:17:24 (1786828644) [ 2222.055322] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:17:31 (1786828651) [ 2228.160291] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:17:38 (1786828658) [ 2233.622589] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:17:43 (1786828663) [ 2239.903764] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:17:50 (1786828670) [ 2247.211474] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:17:57 (1786828677) [ 2255.984502] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:18:05 (1786828685) [ 2265.044518] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:18:14 (1786828694) [ 2273.903144] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:18:23 (1786828703) [ 2281.859364] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:18:31 (1786828711) [ 2288.282806] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:18:38 (1786828718) [ 2288.427931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2288.454703] LNet: 136711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2288.464316] LNetError: 136711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2288.501321] LNet: Removed LNI 192.168.206.16@tcp2 [ 2288.528760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.642845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2288.664975] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2289.813464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2289.835204] LNet: 136855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2289.841558] LNetError: 136855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2289.863556] LNet: Removed LNI 192.168.206.16@tcp [ 2289.887616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2289.952446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2289.984262] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2295.939981] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:18:45 (1786828725) [ 2304.069735] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:18:53 (1786828733) [ 2310.595651] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:19:00 (1786828740) [ 2316.967505] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:19:06 (1786828746) [ 2323.293542] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:19:13 (1786828753) [ 2330.504727] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:19:20 (1786828760) [ 2337.039232] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:19:26 (1786828766) [ 2345.007764] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:19:34 (1786828774) [ 2351.888235] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:19:41 (1786828781) [ 2359.061926] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:19:48 (1786828788) [ 2365.819987] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:19:55 (1786828795) [ 2372.831912] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:20:02 (1786828802) [ 2379.756906] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2181 sec ========== 17:20:09 (1786828809) [ 2380.044832] LNet: 144321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.055633] LNetError: 144321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2380.073867] LNet: Removed LNI 192.168.206.16@tcp [ 2380.727312] Key type .llcrypt unregistered [ 2380.736709] Key type ._llcrypt unregistered [ 2391.955969] Key type ._llcrypt registered [ 2391.959025] Key type .llcrypt registered [ 2392.302439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2392.313709] alg: No test for adler32 (adler32-zlib) [ 2393.472204] Lustre: Lustre: Build Version: 2.17.57_4_gbf75d46 [ 2393.754391] LNet: Added LNI 192.168.206.16@tcp [8/256/0/180] [ 2395.424172] Key type lgssc registered [ 2396.430681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2460.702825] Lustre: Mounted lustre-client - version 2.17.57_4_gbf75d46 [ 2466.096553] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2474.615736] Lustre: Unmounted lustre-client [ 2510.302673] Key type lgssc unregistered [ 2510.541750] LNet: 147282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.550826] LNetError: 147282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.584960] LNet: Removed LNI 192.168.206.16@tcp [ 2511.260217] Key type .llcrypt unregistered [ 2511.262786] Key type ._llcrypt unregistered