[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 434348316 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002376] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007513] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010131] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.014070] *** VALIDATE selinux *** [ 0.021802] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026279] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028099] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029104] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030102] *** VALIDATE tmpfs *** [ 0.031418] *** VALIDATE proc *** [ 0.032152] *** VALIDATE cgroup *** [ 0.033006] *** VALIDATE cgroup2 *** [ 0.034270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036066] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039024] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042642] debug: unmapping init [mem 0xffffffffb1e59000-0xffffffffb1e60fff] [ 0.045136] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046676] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047020] ... version: 2 [ 0.047956] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.049008] ... value mask: 0000ffffffffffff [ 0.050038] ... max period: 00007fffffffffff [ 0.051009] ... fixed-purpose events: 3 [ 0.051882] ... event mask: 000000070000000f [ 0.053211] rcu: Hierarchical SRCU implementation. [ 0.055305] smp: Bringing up secondary CPUs ... [ 0.056575] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.061016] smp: Brought up 1 node, 4 CPUs [ 0.063018] smpboot: Max logical packages: 1 [ 0.064016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.222026] node 0 deferred pages initialised in 155ms [ 0.224328] devtmpfs: initialized [ 0.225331] x86/mm: Memory block size: 128MB [ 0.228887] gcov: version magic: 0x41383552 [ 0.232367] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.233079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.234254] pinctrl core: initialized pinctrl subsystem [ 0.235207] [ 0.235701] ************************************************************* [ 0.236016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.237013] ** ** [ 0.238019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.239013] ** ** [ 0.240015] ** This means that this kernel is built to expose internal ** [ 0.241012] ** IOMMU data structures, which may compromise security on ** [ 0.242013] ** your system. ** [ 0.243013] ** ** [ 0.244011] ** If you see this message and you are not debugging the ** [ 0.245009] ** kernel, report this immediately to your vendor! ** [ 0.246011] ** ** [ 0.247021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.248009] ************************************************************* [ 0.249749] NET: Registered protocol family 16 [ 0.250000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.251073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.253075] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.256152] cpuidle: using governor menu [ 0.257782] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.260527] PCI: Using configuration type 1 for base access [ 0.262131] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.270080] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.272067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.275157] cryptd: max_cpu_qlen set to 1000 [ 0.278252] ACPI: Added _OSI(Module Device) [ 0.279000] ACPI: Added _OSI(Processor Device) [ 0.281108] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.282015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.286837] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.293255] ACPI: Interpreter enabled [ 0.294069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.295013] ACPI: Using IOAPIC for interrupt routing [ 0.297170] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.301000] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.313619] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.315074] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.317042] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.319131] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.327624] acpiphp: Slot [2] registered [ 0.329173] acpiphp: Slot [5] registered [ 0.330210] acpiphp: Slot [6] registered [ 0.332183] acpiphp: Slot [3] registered [ 0.334134] acpiphp: Slot [4] registered [ 0.335132] acpiphp: Slot [7] registered [ 0.337106] acpiphp: Slot [8] registered [ 0.338094] acpiphp: Slot [9] registered [ 0.340126] acpiphp: Slot [10] registered [ 0.341090] acpiphp: Slot [11] registered [ 0.342098] acpiphp: Slot [12] registered [ 0.344120] acpiphp: Slot [13] registered [ 0.345103] acpiphp: Slot [14] registered [ 0.347094] acpiphp: Slot [15] registered [ 0.348102] acpiphp: Slot [16] registered [ 0.350108] acpiphp: Slot [17] registered [ 0.352129] acpiphp: Slot [18] registered [ 0.353131] acpiphp: Slot [19] registered [ 0.355120] acpiphp: Slot [20] registered [ 0.357169] acpiphp: Slot [21] registered [ 0.358102] acpiphp: Slot [22] registered [ 0.359134] acpiphp: Slot [23] registered [ 0.361125] acpiphp: Slot [24] registered [ 0.362143] acpiphp: Slot [25] registered [ 0.364133] acpiphp: Slot [26] registered [ 0.366170] acpiphp: Slot [27] registered [ 0.367133] acpiphp: Slot [28] registered [ 0.369145] acpiphp: Slot [29] registered [ 0.370132] acpiphp: Slot [30] registered [ 0.372195] acpiphp: Slot [31] registered [ 0.374059] PCI host bridge to bus 0000:00 [ 0.375027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.377020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.379024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.382022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.384020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.386026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.388186] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.390978] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.394139] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.402013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.406067] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.409043] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.411017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.413018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.415640] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.418847] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.421068] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.423889] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.427880] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.437020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.441018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.446041] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.456014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.463015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.474017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.483894] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.490017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.499020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.515023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.526050] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.528466] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.531376] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.533364] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.536363] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.541239] iommu: Default domain type: Passthrough [ 0.543520] SCSI subsystem initialized [ 0.545179] ACPI: bus type USB registered [ 0.546123] usbcore: registered new interface driver usbfs [ 0.548113] usbcore: registered new interface driver hub [ 0.550110] usbcore: registered new device driver usb [ 0.551223] pps_core: LinuxPPS API ver. 1 registered [ 0.553015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.556116] PTP clock support registered [ 0.558236] EDAC MC: Ver: 3.0.0 [ 0.560139] PCI: Using ACPI for IRQ routing [ 0.561831] NetLabel: Initializing [ 0.563015] NetLabel: domain hash size = 128 [ 0.565012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.567076] NetLabel: unlabeled traffic allowed by default [ 0.569116] vgaarb: loaded [ 0.571256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.573018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.580166] clocksource: Switched to clocksource kvm-clock [ 0.692684] VFS: Disk quotas dquot_6.6.0 [ 0.694441] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.697238] *** VALIDATE ramfs *** [ 0.698064] *** VALIDATE hugetlbfs *** [ 0.699269] pnp: PnP ACPI init [ 0.701188] pnp: PnP ACPI: found 6 devices [ 0.719664] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.723094] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.724862] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.727524] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.730374] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.734931] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.737878] NET: Registered protocol family 2 [ 0.740155] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.744663] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.748763] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.754685] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.758317] TCP: Hash tables configured (established 65536 bind 65536) [ 0.761251] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.764309] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.767171] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.770099] NET: Registered protocol family 1 [ 0.773458] RPC: Registered named UNIX socket transport module. [ 0.776144] RPC: Registered udp transport module. [ 0.777832] RPC: Registered tcp transport module. [ 0.779982] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.782638] NET: Registered protocol family 44 [ 0.784497] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.786865] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.789325] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.791895] PCI: CLS 0 bytes, default 64 [ 0.794584] Unpacking initramfs... [ 2.186392] debug: unmapping init [mem 0xffff8cd03cc64000-0xffff8cd03ffcffff] [ 2.190376] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.192343] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.194152] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.722809] Initialise system trusted keyrings [ 2.724601] Key type blacklist registered [ 2.727146] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.736741] zbud: loaded [ 2.740894] *** VALIDATE nfs *** [ 2.742326] *** VALIDATE nfs4 *** [ 2.744263] pstore: using deflate compression [ 2.747472] Platform Keyring initialized [ 2.858656] NET: Registered protocol family 38 [ 2.860430] Key type asymmetric registered [ 2.861895] Asymmetric key parser 'x509' registered [ 2.863875] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.867194] io scheduler mq-deadline registered [ 2.868772] io scheduler kyber registered [ 2.870411] io scheduler bfq registered [ 2.872305] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.875326] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.878030] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.880979] ACPI: Power Button [PWRF] [ 2.886686] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.893558] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.907889] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.936167] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.963058] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.969209] Non-volatile memory driver v1.3 [ 2.970686] Linux agpgart interface v0.103 [ 3.003705] virtio_blk virtio1: [vda] 134472 512-byte logical blocks (68.8 MB/65.7 MiB) [ 3.007111] vda: detected capacity change from 0 to 68849664 [ 3.023358] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.026637] vdb: detected capacity change from 0 to 1073741824 [ 3.033689] libphy: Fixed MDIO Bus: probed [ 3.042127] usbcore: registered new interface driver usbserial_generic [ 3.044900] usbserial: USB Serial support registered for generic [ 3.047354] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.050947] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.052789] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.055178] mousedev: PS/2 mouse device common for all mice [ 3.057875] rtc_cmos 00:05: RTC can wake from S4 [ 3.060346] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.061521] rtc_cmos 00:05: registered as rtc0 [ 3.066752] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.069577] intel_pstate: CPU model not supported [ 3.072230] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.075704] hid: raw HID events driver (C) Jiri Kosina [ 3.079192] usbcore: registered new interface driver usbhid [ 3.079309] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.081314] usbhid: USB HID core driver [ 3.081508] drop_monitor: Initializing network drop monitor service [ 3.088990] Initializing XFRM netlink socket [ 3.091312] NET: Registered protocol family 10 [ 3.094558] Segment Routing with IPv6 [ 3.097117] NET: Registered protocol family 17 [ 3.099270] mpls_gso: MPLS GSO support [ 3.106208] RAS: Correctable Errors collector initialized. [ 3.108723] AVX version of gcm_enc/dec engaged. [ 3.110469] AES CTR mode by8 optimization enabled [ 3.185546] sched_clock: Marking stable (3185463307, 0)->(4081675445, -896212138) [ 3.189352] registered taskstats version 1 [ 3.191350] Loading compiled-in X.509 certificates [ 3.193383] zswap: loaded using pool lzo/zbud [ 3.218718] Key type big_key registered [ 3.230544] Key type encrypted registered [ 3.232059] ima: No TPM chip found, activating TPM-bypass! [ 3.234111] ima: Allocated hash algorithm: sha1 [ 3.235698] ima: No architecture policies found [ 3.237454] evm: Initialising EVM extended attributes: [ 3.239257] evm: security.selinux [ 3.240558] evm: security.ima [ 3.243107] evm: security.capability [ 3.244583] evm: HMAC attrs: 0x1 [ 3.248350] rtc_cmos 00:05: setting system clock to 2026-09-05 08:44:12 UTC (1788597852) [ 3.255762] debug: unmapping init [mem 0xffffffffb2e03000-0xffffffffb2ffffff] [ 3.259024] debug: unmapping init [mem 0xffffffffb1b82000-0xffffffffb1e58fff] [ 3.268059] Write protecting the kernel read-only data: 28672k [ 3.271511] debug: unmapping init [mem 0xffffffffb0203000-0xffffffffb03fffff] [ 3.274350] debug: unmapping init [mem 0xffffffffb0b14000-0xffffffffb0bfffff] [ 3.309365] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.317258] systemd[1]: Detected virtualization kvm. [ 3.318734] systemd[1]: Detected architecture x86-64. [ 3.320641] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.352373] systemd[1]: No hostname configured. [ 3.353765] systemd[1]: Set hostname to . [ 3.355778] random: systemd: uninitialized urandom read (16 bytes read) [ 3.358134] systemd[1]: Initializing machine ID from random generator. [ 3.405918] random: ln: uninitialized urandom read (6 bytes read) [ 3.502568] random: systemd: uninitialized urandom read (16 bytes read) [ 3.505271] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.510235] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.514238] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.156879] device-mapper: uevent: version 1.0.3 [ 4.159303] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.913314] random: fast init done [ 4.921379] virtio_net virtio0 ens2: renamed from eth0 [ 4.962386] scsi host0: ata_piix [ 4.983372] scsi host1: ata_piix [ 4.986057] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.988772] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.607367] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.751044] random: crng init done [ 9.753848] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.245815] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.508574] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.768290] SELinux: Disabled at runtime. [ 11.832816] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.840627] systemd[1]: Detected virtualization kvm. [ 11.842506] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.376630] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.379291] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.383758] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.389292] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.392704] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.401098] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.405192] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 12.476725] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.848679] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.132534] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.203908] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.239308] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.271247] EDAC sbridge: Ver: 1.1.2 [ 14.528614] Key type dns_resolver registered [ 14.840578] NFS: Registering the id_resolver key type [ 14.843051] Key type id_resolver registered [ 14.844779] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg633-client login: [ 68.180563] libcfs: loading out-of-tree module taints kernel. [ 68.308493] Key type ._llcrypt registered [ 68.311734] Key type .llcrypt registered [ 68.967513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 69.000916] alg: No test for adler32 (adler32-zlib) [ 70.499782] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 71.509798] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 73.280175] Key type lgssc registered [ 75.152410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 185.634306] Lustre: Mounted lustre-client [ 190.273673] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 207.438224] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing check_logdir /tmp/testlogs/ [ 211.424358] Lustre: lustre-OST0000-osc-ffff8cd090baf800: disconnect after 23s idle [ 212.034359] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing yml_node [ 216.294434] Lustre: DEBUG MARKER: Client: 2.17.51.74 [ 218.716423] Lustre: DEBUG MARKER: MDS: 2.17.51.74 [ 221.042171] Lustre: DEBUG MARKER: OSS: 2.17.51.74 [ 222.597406] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 04:47:50 EDT 2026 [ 238.089269] LustreError: 7905:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8cd090baf800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 238.120819] Lustre: Unmounted lustre-client [ 273.179663] Key type lgssc unregistered [ 273.441282] LNet: 8353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 273.449262] LNetError: 8353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 274.471480] LNet: Removed LNI 192.168.206.33@tcp [ 275.123229] Key type .llcrypt unregistered [ 275.128907] Key type ._llcrypt unregistered [ 299.931459] Key type ._llcrypt registered [ 299.936028] Key type .llcrypt registered [ 300.166430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 300.194677] alg: No test for adler32 (adler32-zlib) [ 301.133558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 301.203634] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 301.263786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 305.905938] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 306.042604] LNet: 9222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.054893] LNetError: 9222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 306.083054] LNet: Removed LNI 192.168.206.33@tcp [ 306.867772] Key type .llcrypt unregistered [ 306.871244] Key type ._llcrypt unregistered [ 313.408169] Key type ._llcrypt registered [ 313.409847] Key type .llcrypt registered [ 314.889473] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:49:22 (1788598162) [ 315.247860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 315.255971] alg: No test for adler32 (adler32-zlib) [ 316.163120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.274956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.299815] LNet: 9642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.337712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.444528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 316.460659] LNet: 9649:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 322.263267] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:49:30 (1788598170) [ 322.430729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.481975] LNet: 10225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.516441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.579507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 328.987013] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:49:36 (1788598176) [ 329.240028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.280121] LNet: 10810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.324250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.398990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 335.541618] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:49:43 (1788598183) [ 335.687025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.704270] LNet: 11392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.734297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.820947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 342.531527] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:49:50 (1788598190) [ 342.633311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.669421] LNet: 11975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.696395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.798838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 343.359550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 351.264436] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:49:58 (1788598198) [ 351.371719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.405618] LNet: 12566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.438118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.520685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 352.026624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 352.633586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 359.008720] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:50:07 (1788598207) [ 359.156178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.185857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.236431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 365.050541] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:50:13 (1788598213) [ 365.385681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.449927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.518365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 365.544784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 366.161426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 366.222220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 366.844931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 366.911955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 367.404698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 367.437560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 367.833761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 367.861849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 368.159493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 368.196270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 368.529797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 368.562838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 369.174724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 369.240828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 369.763803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 369.800452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 370.271626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 370.310872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 370.720450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 370.756812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 371.151509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 371.197896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 371.557865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 371.592294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 379.038496] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:50:26 (1788598226) [ 379.127915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.203696] LNet: 14429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.209923] LNet: 14429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 379.250302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.405607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 379.492865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 386.325100] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:50:34 (1788598234) [ 386.471893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.504916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.590609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 386.626087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 392.807312] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:50:40 (1788598240) [ 393.085838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.146840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.233520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 393.283352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 396.522366] hrtimer: interrupt took 7660764 ns [ 399.700681] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:50:47 (1788598247) [ 400.025442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.098630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.244525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 400.282726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 407.581495] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:50:55 (1788598255) [ 407.666287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.767663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.870444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 407.920467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 414.494772] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:51:02 (1788598262) [ 414.800167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.837940] LNet: 17367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.848680] LNet: 17367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 414.879830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.978606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 415.019686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 421.431446] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:51:09 (1788598269) [ 421.702641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.751980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.860252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 422.186356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 429.208361] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:51:17 (1788598277) [ 429.307861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.369337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.566120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 429.754836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 436.106486] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:51:24 (1788598284) [ 436.170729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.207447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.329902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.527170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 442.611554] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:51:30 (1788598290) [ 442.989292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.045614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.502027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 443.746985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 450.158843] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:51:38 (1788598298) [ 450.307979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.387843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.524869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.590575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 456.567178] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:51:44 (1788598304) [ 456.974554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.012664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.100374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 457.164269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.120784] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:51:52 (1788598312) [ 464.244358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.298303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.378038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 470.930664] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:51:58 (1788598318) [ 471.121217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.182371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.302105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 477.690474] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:52:05 (1788598325) [ 478.090721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.133586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.228469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 478.944871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 485.603560] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:52:13 (1788598333) [ 485.853665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.882710] LNet: 23393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.899774] LNet: 23393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 485.981500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.095222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 486.151342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 492.298793] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:52:20 (1788598340) [ 492.533950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.589189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.704039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 492.747520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 499.096520] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:52:26 (1788598346) [ 499.185583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.237187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.350482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 499.403959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 505.591183] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:52:33 (1788598353) [ 505.776860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.859842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.960637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 506.004656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 512.195257] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:52:40 (1788598360) [ 512.301091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.361139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.426851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 512.483254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 512.509443] LNetError: 25751:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 512.556122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 512.586766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 517.905704] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:52:45 (1788598365) [ 518.004792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.055142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.136936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.1O8qNNrGvo [ 518.232969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.1O8qNNrGvo [ 518.288173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.1O8qNNrGvo [ 523.990358] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:52:51 (1788598371) [ 524.083783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.133451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.207038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 524.280149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 524.331211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 530.483452] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:52:58 (1788598378) [ 530.809286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.838900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.921650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 534.148891] LNet: Message forwarding will be enabled in 2s [ 537.251997] LNet: Message forwarding enabled [ 537.588181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 537.616338] LNet: Message forwarding will stop in 2s to 6s [ 542.140930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 542.724075] LNet: Message forwarding disabled [ 544.661626] LNet: Message forwarding will be enabled in 2s [ 547.872174] LNet: Message forwarding enabled [ 551.339142] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:53:19 (1788598399) [ 551.645772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.882978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.934153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 558.174993] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:53:26 (1788598406) [ 558.300947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.366894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.430520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 558.459824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 558.495556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 558.526534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 558.550903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 558.590957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 558.629848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 558.677726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 558.707743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 558.757641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 558.801252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 558.860382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 558.901537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 558.957525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 559.011420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 559.056807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 559.116624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 559.168697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 559.214077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 559.246783] LNetError: 28823:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 565.363908] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:53:33 (1788598413) [ 565.478093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.559100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.694856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 572.355891] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:53:40 (1788598420) [ 574.807989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg633-server [ 576.836591] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:53:44 (1788598424) [ 576.967511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.045416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.144383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 577.183639] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 579.460194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[34-38]@tcp [ 579.489172] LNet: 30539:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 579.551202] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 579.574337] LNetError: 30326:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.34@tcp has gone from up to down [ 580.521852] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.38@tcp is being used as a gateway but routing feature is not turned on [ 580.535505] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 581.548791] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 581.577249] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 583.630116] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 583.654479] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 586.602545] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:53:54 (1788598434) [ 587.706325] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 587.738903] LNetError: 30326:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 589.323986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg633-server [ 591.755065] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:53:58 (1788598438) [ 591.885299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.917145] LNetError: 31468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.939182] LNet: Removed LNI 192.168.206.33@tcp [ 591.972051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.069205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.140415] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 593.299977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[34-39/2]@tcp [ 593.327494] LNet: 31591:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 593.339238] LNet: 31591:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 593.376764] LNetError: 31476:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.36@tcp has gone from up to down [ 593.379294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[34-39/2]@tcp [ 593.389733] LNetError: 31476:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 602.152772] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:54:09 (1788598449) [ 602.307797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.351383] LNetError: 32177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.370090] LNet: Removed LNI 192.168.206.33@tcp [ 602.407262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.527269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 602.590804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.631682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.692375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 602.781668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.846555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.944631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 603.082807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.149761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.268625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 603.402803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.489841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.595353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 603.733643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.812648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.895503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 604.016558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.109509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.211363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 604.352412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.399254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.520868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 604.678436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.758659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.824496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 604.939543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.989897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.068469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.155704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.189060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.249598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 605.313264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.361604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.451139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 613.545135] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:54:21 (1788598461) [ 616.382847] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg633-server [ 618.094859] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:54:25 (1788598465) [ 620.266585] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg633-server [ 621.555290] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:54:29 (1788598469) [ 621.823544] LNet: 33585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.844933] LNet: 33585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 24 previous similar messages [ 622.568781] Key type .llcrypt unregistered [ 622.573947] Key type ._llcrypt unregistered [ 629.665042] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 629.936861] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 629.946178] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 629.951596] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 632.832486] Key type ._llcrypt registered [ 632.838513] Key type .llcrypt registered [ 633.016467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 633.027864] alg: No test for adler32 (adler32-zlib) [ 633.951399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.012485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 634.033253] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 634.740788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 634.766123] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 636.495992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 636.546321] LNet: Removed LNI 10.1.2.3@tcp [ 636.720682] LNet: 34300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.744389] LNetError: 34300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.781524] LNet: Removed LNI 192.168.206.33@tcp [ 637.432049] Key type .llcrypt unregistered [ 637.436711] Key type ._llcrypt unregistered [ 648.701796] Key type ._llcrypt registered [ 648.704658] Key type .llcrypt registered [ 650.416222] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:54:57 (1788598497) [ 651.054198] Key type .llcrypt unregistered [ 651.056939] Key type ._llcrypt unregistered [ 658.036744] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 658.261585] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 658.289344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 658.302842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 661.112818] Key type ._llcrypt registered [ 661.123582] Key type .llcrypt registered [ 661.401924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 661.411231] alg: No test for adler32 (adler32-zlib) [ 662.339237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.430806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 662.452954] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 663.220462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 663.238586] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 666.189710] LNet: 35787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.206109] LNetError: 35787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.220408] LNet: Removed LNI 192.168.206.33@tcp [ 666.231118] LNet: Removed LNI 10.1.2.3@tcp [ 667.265781] Key type .llcrypt unregistered [ 667.271991] Key type ._llcrypt unregistered [ 679.528763] Key type ._llcrypt registered [ 679.530850] Key type .llcrypt registered [ 680.819823] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:55:28 (1788598528) [ 681.412402] Key type .llcrypt unregistered [ 681.420650] Key type ._llcrypt unregistered [ 686.633268] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 686.746908] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 686.751504] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 686.754403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 689.670909] Key type ._llcrypt registered [ 689.679905] Key type .llcrypt registered [ 690.255334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 690.297238] alg: No test for adler32 (adler32-zlib) [ 691.394257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.498149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 691.539254] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 694.403569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 694.441375] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 697.232666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 697.260376] LNet: Removed LNI 10.1.2.3@tcp [ 697.281442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 697.325504] LNet: Removed LNI 10.1.2.31@tcp [ 697.355054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 697.406089] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 698.283645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 698.302969] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 699.528438] LNet: 37703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 699.535836] LNetError: 37703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 699.541810] LNet: Removed LNI 10.1.2.3@tcp [ 699.567385] LNet: Removed LNI 10.1.2.31@tcp [ 700.293805] Key type .llcrypt unregistered [ 700.301300] Key type ._llcrypt unregistered [ 712.650334] Key type ._llcrypt registered [ 712.653081] Key type .llcrypt registered [ 714.190920] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:56:01 (1788598561) [ 714.623408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 714.636165] alg: No test for adler32 (adler32-zlib) [ 715.643490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.728947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 715.780766] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 716.977883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.042399] LNet: 38658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.058253] LNetError: 38658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.072030] LNet: Removed LNI 192.168.206.33@tcp [ 717.098307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.172650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.33@tcp [ 717.225230] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 718.110688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.33@tcp [ 718.143992] LNet: Removed LNI 192.168.206.33@tcp [ 718.251654] LNet: 38793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.327377] Key type .llcrypt unregistered [ 719.330289] Key type ._llcrypt unregistered [ 731.539487] Key type ._llcrypt registered [ 731.541219] Key type .llcrypt registered [ 732.561841] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:56:20 (1788598580) [ 734.485969] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg633-server [ 736.150521] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:56:24 (1788598584) [ 736.647334] Key type .llcrypt unregistered [ 736.653991] Key type ._llcrypt unregistered [ 742.245322] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 742.356024] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 742.359468] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 742.364190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 745.175452] Key type ._llcrypt registered [ 745.181673] Key type .llcrypt registered [ 745.431651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 745.450858] alg: No test for adler32 (adler32-zlib) [ 746.652235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.766081] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 747.963598] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 751.177297] LNet: 40629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.194841] LNetError: 40629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.204202] LNet: Removed LNI 192.168.206.33@tcp1 [ 751.220447] LNet: Removed LNI 10.1.2.3@tcp2 [ 752.408832] Key type .llcrypt unregistered [ 752.427526] Key type ._llcrypt unregistered [ 765.229472] Key type ._llcrypt registered [ 765.231722] Key type .llcrypt registered [ 766.524559] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:56:54 (1788598614) [ 766.928417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 766.941708] alg: No test for adler32 (adler32-zlib) [ 767.988632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.043741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 768.073763] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 768.951290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.977644] LNet: 41599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.987344] LNetError: 41599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.008783] LNet: Removed LNI 192.168.206.33@tcp [ 769.031611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.096459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 769.116631] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 776.454231] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:57:04 (1788598624) [ 776.665685] LNet: 42190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.687414] LNetError: 42190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.703304] LNet: Removed LNI 192.168.206.33@tcp [ 777.450627] Key type .llcrypt unregistered [ 777.453447] Key type ._llcrypt unregistered [ 783.535796] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 783.687582] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 783.694426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 783.706723] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 786.209663] Key type ._llcrypt registered [ 786.212191] Key type .llcrypt registered [ 786.392520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 786.420584] alg: No test for adler32 (adler32-zlib) [ 787.374977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.441369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 787.459678] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 788.118499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 788.152891] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 789.938824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 789.967164] LNet: 42933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.990484] LNetError: 42933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.006576] LNet: Removed LNI 192.168.206.33@tcp [ 790.026713] LNet: Removed LNI 10.1.2.3@tcp [ 790.062286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.135805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 790.179917] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 790.188454] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.738479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.760718] LNet: 42960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.768344] LNetError: 42960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.782071] LNetError: 42960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 790.790668] LNet: Removed LNI 192.168.206.33@tcp [ 790.809145] LNet: Removed LNI 10.1.2.3@tcp [ 790.833542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.914692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 790.937547] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 790.942078] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 791.516780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 791.557878] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 792.567836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.606728] LNet: 43122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.612819] LNetError: 43122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.617744] LNetError: 43122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 792.626274] LNet: Removed LNI 192.168.206.33@tcp [ 792.630532] LNet: Removed LNI 10.1.2.3@tcp [ 792.672723] LNet: Removed LNI 192.168.206.33@tcp2 [ 792.712918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.804809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 792.842661] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 792.851191] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.856891] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 800.493855] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:57:28 (1788598648) [ 800.898213] LNet: 43720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.915858] LNetError: 43720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.921237] LNetError: 43720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 800.927262] LNet: Removed LNI 192.168.206.33@tcp [ 800.930938] LNet: Removed LNI 10.1.2.3@tcp [ 800.950167] LNet: Removed LNI 192.168.206.33@tcp2 [ 801.724064] Key type .llcrypt unregistered [ 801.729773] Key type ._llcrypt unregistered [ 809.926629] Key type ._llcrypt registered [ 809.928664] Key type .llcrypt registered [ 810.129571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.150978] alg: No test for adler32 (adler32-zlib) [ 811.081800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.140845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 811.165891] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 812.211755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 812.246758] LNet: 44115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.256305] LNetError: 44115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.270221] LNet: Removed LNI 192.168.206.33@tcp [ 812.295988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.343661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 812.363503] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 813.041049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.110644] LNet: 44145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.125658] LNetError: 44145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.145884] LNet: Removed LNI 192.168.206.33@tcp [ 813.192204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.313149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 813.382536] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 813.924297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.980344] LNet: Removed LNI 192.168.206.33@tcp [ 814.036705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.109593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 814.136841] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 814.598277] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 814.803476] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 814.811870] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 814.816351] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.115806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.138975] LNet: 44254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.144529] LNet: 44254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 817.151227] LNetError: 44254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.155612] LNetError: 44254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 817.176970] LNet: Removed LNI 192.168.206.33@tcp [ 817.194532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.269980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 817.298205] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 823.427903] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:57:51 (1788598671) [ 823.648524] LNet: 44851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.658738] LNetError: 44851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.677564] LNet: Removed LNI 192.168.206.33@tcp [ 824.250775] Key type .llcrypt unregistered [ 824.256372] Key type ._llcrypt unregistered [ 830.022616] Key type ._llcrypt registered [ 830.028419] Key type .llcrypt registered [ 830.248142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 830.257964] alg: No test for adler32 (adler32-zlib) [ 831.146397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.189827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 831.207059] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 831.992847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.007923] LNet: 45250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.012467] LNetError: 45250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.028656] LNet: Removed LNI 192.168.206.33@tcp [ 832.048385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.113313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 832.149430] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 838.534312] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:58:06 (1788598686) [ 838.765358] LNet: 45843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.775457] LNetError: 45843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.787823] LNet: Removed LNI 192.168.206.33@tcp [ 839.350622] Key type .llcrypt unregistered [ 839.353941] Key type ._llcrypt unregistered [ 844.665964] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 844.809275] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 844.812609] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 844.818344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 847.445198] Key type ._llcrypt registered [ 847.449058] Key type .llcrypt registered [ 847.677331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 847.707590] alg: No test for adler32 (adler32-zlib) [ 848.628365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.717985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.749818] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 849.628610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.664222] LNet: 46287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.682154] LNetError: 46287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.709208] LNet: Removed LNI 192.168.206.33@tcp [ 849.728771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.773754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 849.814285] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 855.618189] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:58:23 (1788598703) [ 855.722566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.743071] LNet: 46877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.755094] LNetError: 46877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.782423] LNet: Removed LNI 192.168.206.33@tcp [ 855.801976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.900873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 861.202100] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:58:29 (1788598709) [ 861.288473] LNet: 47460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.821671] Key type .llcrypt unregistered [ 861.823420] Key type ._llcrypt unregistered [ 867.243407] Key type ._llcrypt registered [ 867.245342] Key type .llcrypt registered [ 867.468478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 867.488803] alg: No test for adler32 (adler32-zlib) [ 868.391362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.441109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 868.466599] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 869.226241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.250741] LNet: 47864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.256978] LNetError: 47864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.278454] LNet: Removed LNI 192.168.206.33@tcp [ 869.307665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.363044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 869.381267] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 875.500761] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:58:43 (1788598723) [ 875.746829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.771107] LNet: 48457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.776513] LNetError: 48457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.790897] LNet: Removed LNI 192.168.206.33@tcp [ 875.817586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.921661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 882.071235] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:58:50 (1788598730) [ 883.544501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 885.234442] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:58:53 (1788598733) [ 886.611165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 887.862263] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:58:56 (1788598736) [ 889.116905] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 890.388404] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:58:58 (1788598738) [ 891.732747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 893.320085] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:59:01 (1788598741) [ 894.729638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 896.193850] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:59:04 (1788598744) [ 896.548490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.578835] LNet: 50509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.601524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.653994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.685153] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 897.514963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.33 [ 897.537250] LNet: 50642:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 897.543770] LNet: Removed LNI 192.168.206.33@tcp [ 904.030631] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:59:11 (1788598751) [ 904.257649] LNet: 51215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.271556] LNetError: 51215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.289701] LNet: Removed LNI 192.168.206.33@tcp [ 904.892888] Key type .llcrypt unregistered [ 904.895927] Key type ._llcrypt unregistered [ 910.589106] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.851766] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.858870] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.869350] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.547954] Key type ._llcrypt registered [ 912.550456] Key type .llcrypt registered [ 912.693123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.708635] alg: No test for adler32 (adler32-zlib) [ 914.002978] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 920.548358] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:59:28 (1788598768) [ 920.826789] LNet: 52112:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.838334] LNetError: 52112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.864706] LNet: Removed LNI 10.1.2.3@tcp [ 921.509570] Key type .llcrypt unregistered [ 921.511853] Key type ._llcrypt unregistered [ 927.758639] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 928.033238] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 928.038428] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 928.049788] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 930.665787] Key type ._llcrypt registered [ 930.669894] Key type .llcrypt registered [ 930.897824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 930.912500] alg: No test for adler32 (adler32-zlib) [ 931.866640] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 937.822822] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:59:45 (1788598785) [ 938.032719] LNet: 53005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.038799] LNetError: 53005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 938.054034] LNet: Removed LNI 10.1.2.3@tcp [ 938.563559] Key type .llcrypt unregistered [ 938.571073] Key type ._llcrypt unregistered [ 944.179746] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 944.435606] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 944.439524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 944.442775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 947.253689] Key type ._llcrypt registered [ 947.255700] Key type .llcrypt registered [ 947.395921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 947.415194] alg: No test for adler32 (adler32-zlib) [ 948.373669] LNet: 53305:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 948.385437] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 954.961792] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 05:00:02 (1788598802) [ 955.044338] LNet: 53889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.056240] LNetError: 53889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 955.071870] LNet: Removed LNI 10.1.2.3@tcp [ 955.851518] Key type .llcrypt unregistered [ 955.858848] Key type ._llcrypt unregistered [ 960.715954] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 961.051169] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 961.058170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 961.069668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 963.781678] Key type ._llcrypt registered [ 963.783992] Key type .llcrypt registered [ 964.048916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.092391] alg: No test for adler32 (adler32-zlib) [ 965.037544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.135406] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 973.647142] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 05:00:21 (1788598821) [ 973.960247] LNet: 55126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.968435] LNetError: 55126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 973.984377] LNet: Removed LNI 10.1.2.3@tcp [ 974.508566] Key type .llcrypt unregistered [ 974.510958] Key type ._llcrypt unregistered [ 979.375568] Key type ._llcrypt registered [ 979.378415] Key type .llcrypt registered [ 979.532528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 979.559556] alg: No test for adler32 (adler32-zlib) [ 980.488974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 980.523276] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 987.205525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 988.105790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_interrupt -r 1 [ 988.147523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_interrupt -r 1 [ 988.184808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 988.211430] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 988.456764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 988.492184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 988.525593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 988.554938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 988.580626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 988.610937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 988.644889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 988.685797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 988.704459] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 988.704685] LNet: 55564:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 988.735539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 988.822618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 988.863615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 988.892444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 988.945985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.991200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 989.025663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 989.582046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_dropped -r 1 [ 989.632957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_dropped -r 1 [ 989.678270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 989.705660] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 989.879763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 989.908535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 989.937729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 989.962369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 989.988495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 990.019250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 990.046662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 990.069529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 990.089760] LNet: 55673:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 990.104398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 990.133633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 990.163935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.195634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 990.227686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 990.259518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 990.284439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 990.774991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_aborted -r 1 [ 990.828657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_aborted -r 1 [ 990.901359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 990.931862] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 990.939877] LNet: Skipped 1 previous similar message [ 991.185299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 991.228057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 991.263426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 991.305857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 991.339163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 991.386462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 991.415361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 991.450471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 991.480058] LNet: 55782:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 991.508663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 991.544031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 991.576603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 991.605607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 991.637485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 991.669863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 991.714550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 992.195849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_no_route -r 1 [ 992.230893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_no_route -r 1 [ 992.260840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 992.482387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 992.511386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 992.536274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 992.557905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 992.592406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.616674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.642646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.664642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.699771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 992.724535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 992.754804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 992.792287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 992.838749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 992.873053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 992.914362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 993.457887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_timeout -r 1 [ 993.500671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_timeout -r 1 [ 993.534762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 993.553710] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 993.558604] LNet: Skipped 1 previous similar message [ 993.781699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 993.811027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 993.851814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 993.905486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 993.939930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 993.966952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 994.000384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 994.036144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 994.051996] LNet: 56001:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 994.057257] LNet: 56001:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 994.073124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 994.111952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 994.140177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 994.169551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 994.195565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 994.221436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 994.252390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 994.785525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_error -r 1 [ 994.830942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_error -r 1 [ 994.879335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 995.147643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 995.173981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 995.208350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 995.237449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 995.267171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 995.303514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 995.330684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 995.362291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 995.389512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 995.420997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 995.472073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 995.510374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 995.552809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 995.598893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 995.649270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 995.904887] LNet: 56151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.911924] LNetError: 56151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.971816] LNet: Removed LNI 192.168.206.33@tcp [ 997.341660] Key type .llcrypt unregistered [ 997.345719] Key type ._llcrypt unregistered [ 1012.906879] Key type ._llcrypt registered [ 1012.914875] Key type .llcrypt registered [ 1014.422263] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 05:01:01 (1788598861) [ 1015.321462] Key type .llcrypt unregistered [ 1015.324807] Key type ._llcrypt unregistered [ 1021.368175] Key type ._llcrypt registered [ 1021.375967] Key type .llcrypt registered [ 1021.591141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.606647] alg: No test for adler32 (adler32-zlib) [ 1022.478461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1022.505546] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1032.037195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1043.840166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.871145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1043.899376] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1045.351511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_interrupt -r 1 [ 1045.455929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_interrupt -r 1 [ 1045.510098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_interrupt -r 1 [ 1045.571092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_interrupt -r 1 [ 1045.620210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_interrupt -r 1 [ 1045.681296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_interrupt -r 1 [ 1045.779436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_interrupt -r 1 [ 1045.838737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_interrupt -r 1 [ 1045.887340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1045.912905] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1045.918521] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1046.191292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.219256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1046.246797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.271785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.298441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.327643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.358240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.383995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.397627] LNet: 57634:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1046.417914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.449417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1046.489773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1046.524340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1046.591326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1046.649461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1046.706194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1047.354669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_dropped -r 1 [ 1047.412964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_dropped -r 1 [ 1047.464285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_dropped -r 1 [ 1047.499151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_dropped -r 1 [ 1047.530150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_dropped -r 1 [ 1047.580273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_dropped -r 1 [ 1047.616272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_dropped -r 1 [ 1047.668143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_dropped -r 1 [ 1047.714554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1047.739359] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1047.743799] LNet: Skipped 2 previous similar messages [ 1047.952919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.024770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1048.071775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.144441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.216549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.271582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.329546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.356317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.367871] LNet: 57756:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1048.397230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.420061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.448071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.479559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.497994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1048.521544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1048.548361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1049.128960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_aborted -r 1 [ 1049.202230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_aborted -r 1 [ 1049.257640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_aborted -r 1 [ 1049.333936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_aborted -r 1 [ 1049.408273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_aborted -r 1 [ 1049.460046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_aborted -r 1 [ 1049.542985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_aborted -r 1 [ 1049.584737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_aborted -r 1 [ 1049.606413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1049.618628] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1049.623201] LNet: Skipped 2 previous similar messages [ 1049.839576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1049.902671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1049.977488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.042425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.103861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.170832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.267990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.345248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.379822] LNet: 57877:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1050.407716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.442532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.479949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.548147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.589274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.635657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.691213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1051.514296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_no_route -r 1 [ 1051.578243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_no_route -r 1 [ 1051.634530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_no_route -r 1 [ 1051.685458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_no_route -r 1 [ 1051.709857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_no_route -r 1 [ 1051.773124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_no_route -r 1 [ 1051.838060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_no_route -r 1 [ 1051.896653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_no_route -r 1 [ 1051.932665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1051.948710] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1051.952985] LNet: Skipped 4 previous similar messages [ 1052.091439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.116778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1052.158430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.181642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.201532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.235785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.262662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.294173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1052.332471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1052.356992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1052.397278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1052.441782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1052.469167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.500772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1052.534072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1052.877564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_timeout -r 1 [ 1052.904828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_timeout -r 1 [ 1052.947422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_timeout -r 1 [ 1052.990325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_timeout -r 1 [ 1053.011295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_timeout -r 1 [ 1053.030793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_timeout -r 1 [ 1053.055373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_timeout -r 1 [ 1053.087431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_timeout -r 1 [ 1053.123503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1053.281085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.306657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1053.329907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.357676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.381731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.407327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.437897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.464928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.489499] LNet: 58120:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.495818] LNet: 58120:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1053.521624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.562392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.598800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.629605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.656288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.677965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.705759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1054.143026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_error -r 1 [ 1054.192857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_error -r 1 [ 1054.217945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_error -r 1 [ 1054.247143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_error -r 1 [ 1054.288862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_error -r 1 [ 1054.336759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_error -r 1 [ 1054.373185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_error -r 1 [ 1054.436220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_error -r 1 [ 1054.465150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1054.663553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1054.691556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1054.716335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1054.737643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1054.765965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1054.786096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1054.842142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1054.865292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1054.903775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1054.945334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1054.978743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1055.032054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1055.069995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1055.093121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1055.113326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1060.453791] LNet: 58317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.458659] LNetError: 58317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.464238] LNet: Removed LNI 192.168.206.33@tcp [ 1061.480807] LNet: Removed LNI 192.168.206.33@tcp1 [ 1061.915968] Key type .llcrypt unregistered [ 1061.922749] Key type ._llcrypt unregistered [ 1079.654213] Key type ._llcrypt registered [ 1079.655964] Key type .llcrypt registered [ 1080.949733] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 05:02:08 (1788598928) [ 1082.230562] Key type .llcrypt unregistered [ 1082.236385] Key type ._llcrypt unregistered [ 1089.885328] Key type ._llcrypt registered [ 1089.887591] Key type .llcrypt registered [ 1090.088300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.107916] alg: No test for adler32 (adler32-zlib) [ 1091.072969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1091.134951] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1099.336422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1099.898747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_dropped -r 1 [ 1099.924667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_dropped -r 1 [ 1099.949486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1099.971371] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1100.164578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.190455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1100.215786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.250444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.275812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.302847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.335301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.362599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.378735] LNet: 59568:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.392595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.412555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.442663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.477778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.501247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.539795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.608536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1101.067972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_error -r 1 [ 1101.112220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_error -r 1 [ 1101.157586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1101.179817] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1101.462934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.502409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1101.537477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.565113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1101.596107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1101.624404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1101.655977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1101.678596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1101.690727] LNet: 59677:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1101.704960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1101.722952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.755915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.799651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.826400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.867348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.910032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1102.469870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_timeout -r 1 [ 1102.506192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_timeout -r 1 [ 1102.551476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1102.574423] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1102.804389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.841522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1102.884525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.920909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.971847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.030514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.089884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.140714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.154760] LNet: 59787:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.182642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.245759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.309071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.356560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1103.408664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1103.459705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1103.509523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1103.832226] LNet: 59828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.837747] LNetError: 59828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.851743] LNet: Removed LNI 192.168.206.33@tcp [ 1104.190562] Key type .llcrypt unregistered [ 1104.193658] Key type ._llcrypt unregistered [ 1118.342542] Key type ._llcrypt registered [ 1118.346217] Key type .llcrypt registered [ 1119.559758] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 05:02:47 (1788598967) [ 1120.398889] Key type .llcrypt unregistered [ 1120.401406] Key type ._llcrypt unregistered [ 1125.288412] Key type ._llcrypt registered [ 1125.290045] Key type .llcrypt registered [ 1125.526974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1125.543787] alg: No test for adler32 (adler32-zlib) [ 1126.440898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1126.465106] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1132.645519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1139.088874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1139.135953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1139.173257] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1140.390360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_dropped -r 1 [ 1140.421561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e remote_dropped -r 1 [ 1140.450129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_dropped -r 1 [ 1140.478671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e remote_dropped -r 1 [ 1140.535256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e remote_dropped -r 1 [ 1140.577176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e remote_dropped -r 1 [ 1140.610720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e remote_dropped -r 1 [ 1140.649968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e remote_dropped -r 1 [ 1140.686226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1140.702621] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1140.708694] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.133@tcp [ 1140.912235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.952150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1140.988552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1141.026839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1141.047248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1141.077625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1141.099320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1141.122216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1141.143675] LNet: 61297:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1141.157969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1141.184463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1141.217802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1141.243975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1141.272827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.303387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.339616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1141.372654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.783914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_error -r 1 [ 1141.813958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e remote_error -r 1 [ 1141.846940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_error -r 1 [ 1141.882262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e remote_error -r 1 [ 1141.918691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e remote_error -r 1 [ 1141.961220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e remote_error -r 1 [ 1142.009193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e remote_error -r 1 [ 1142.057168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e remote_error -r 1 [ 1142.089575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1142.104769] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1142.111262] LNet: Skipped 2 previous similar messages [ 1142.300776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.328536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1142.350391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.378726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.410610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.450832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.478915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.508944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.526604] LNet: 61420:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1142.545309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.589828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.629519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.672523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.706246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1142.747527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1142.783980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1142.821712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1143.216787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_timeout -r 1 [ 1143.269775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e remote_timeout -r 1 [ 1143.321854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_timeout -r 1 [ 1143.354374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e remote_timeout -r 1 [ 1143.395973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e remote_timeout -r 1 [ 1143.430710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e remote_timeout -r 1 [ 1143.460470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e remote_timeout -r 1 [ 1143.499843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e remote_timeout -r 1 [ 1143.549286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1143.575772] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1143.801890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.835724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1143.886600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1143.923179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1143.960705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.010981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.055853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.075321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.092276] LNet: 61543:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1144.111223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.138708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.167343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.193848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.224897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.247625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.275423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1144.301917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.179627] LNet: 61621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.190120] LNetError: 61621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.196303] LNet: Removed LNI 192.168.206.33@tcp [ 1148.215492] LNet: Removed LNI 192.168.206.33@tcp1 [ 1148.549277] Key type .llcrypt unregistered [ 1148.556421] Key type ._llcrypt unregistered [ 1164.454694] Key type ._llcrypt registered [ 1164.460697] Key type .llcrypt registered [ 1166.477382] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 05:03:33 (1788599013) [ 1167.181278] Key type .llcrypt unregistered [ 1167.190976] Key type ._llcrypt unregistered [ 1174.362853] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1174.699961] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1174.707047] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1174.715288] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1177.245628] Key type ._llcrypt registered [ 1177.252522] Key type .llcrypt registered [ 1177.401354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1177.431111] alg: No test for adler32 (adler32-zlib) [ 1178.403071] LNet: 62699:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1178.432193] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1178.510030] LNet: 62713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.522395] LNetError: 62713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.566861] LNet: Removed LNI 192.168.206.33@tcp [ 1179.314792] Key type .llcrypt unregistered [ 1179.322339] Key type ._llcrypt unregistered [ 1184.907197] Key type ._llcrypt registered [ 1184.911024] Key type .llcrypt registered [ 1185.110669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1185.120771] alg: No test for adler32 (adler32-zlib) [ 1186.016312] LNet: 62972:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1186.038757] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1186.042694] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1186.080210] LNet: 62986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.089487] LNetError: 62986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.098858] LNet: Removed LNI 192.168.206.33@tcp [ 1186.109243] LNet: Removed LNI 10.1.2.3@tcp1 [ 1186.669641] Key type .llcrypt unregistered [ 1186.672419] Key type ._llcrypt unregistered [ 1191.695081] Key type ._llcrypt registered [ 1191.699053] Key type .llcrypt registered [ 1191.876658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.902878] alg: No test for adler32 (adler32-zlib) [ 1192.861370] LNet: 63246:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1192.874140] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1192.876903] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1192.915275] LNet: 63260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.920589] LNetError: 63260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.924769] LNet: Removed LNI 192.168.206.33@tcp [ 1192.931590] LNet: Removed LNI 10.1.2.3@tcp [ 1193.628527] Key type .llcrypt unregistered [ 1193.630624] Key type ._llcrypt unregistered [ 1198.961342] Key type ._llcrypt registered [ 1198.965068] Key type .llcrypt registered [ 1199.201513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1199.211906] alg: No test for adler32 (adler32-zlib) [ 1200.176860] LNet: 63520:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1200.192839] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1200.197032] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1200.250844] LNet: 63534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.258223] LNetError: 63534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.263488] LNet: Removed LNI 192.168.206.33@tcp [ 1200.272165] LNet: Removed LNI 10.1.2.3@tcp [ 1200.972487] Key type .llcrypt unregistered [ 1200.977818] Key type ._llcrypt unregistered [ 1206.353224] Key type ._llcrypt registered [ 1206.356284] Key type .llcrypt registered [ 1206.556654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1206.570208] alg: No test for adler32 (adler32-zlib) [ 1207.478493] LNet: 63799:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1207.494738] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1207.500101] LNetError: 63799:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1207.505677] LNetError: Error -100 starting up LNI tcp [ 1207.517176] LNet: Removed LNI 192.168.206.33@tcp [ 1208.159408] Key type .llcrypt unregistered [ 1208.162881] Key type ._llcrypt unregistered [ 1218.783661] Key type ._llcrypt registered [ 1218.786384] Key type .llcrypt registered [ 1220.004859] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 05:04:27 (1788599067) [ 1220.909699] Key type .llcrypt unregistered [ 1220.912314] Key type ._llcrypt unregistered [ 1226.702566] Key type ._llcrypt registered [ 1226.704474] Key type .llcrypt registered [ 1226.858432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1226.872289] alg: No test for adler32 (adler32-zlib) [ 1227.828660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1227.881243] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1235.262727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1235.800767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e network_timeout -r 1 [ 1235.826200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e network_timeout -r 1 [ 1235.855694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1235.872162] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1236.038023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.067252] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1236.076678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1236.121967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.144892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.176257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.217707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.244952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.268939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.283410] LNet: 65054:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.296211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.329963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.357813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.386363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.424608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.449056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.727992] LNet: 65094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.735364] LNetError: 65094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.750206] LNet: Removed LNI 192.168.206.33@tcp [ 1237.201847] Key type .llcrypt unregistered [ 1237.206326] Key type ._llcrypt unregistered [ 1252.350616] Key type ._llcrypt registered [ 1252.352873] Key type .llcrypt registered [ 1252.514596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1252.531954] alg: No test for adler32 (adler32-zlib) [ 1253.411396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1253.434796] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1259.441820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1267.721776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1267.773917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1267.796671] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1269.062916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e network_timeout -r 1 [ 1269.100528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e network_timeout -r 1 [ 1269.154228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e network_timeout -r 1 [ 1269.207840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e network_timeout -r 1 [ 1269.250800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e network_timeout -r 1 [ 1269.326180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e network_timeout -r 1 [ 1269.372949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e network_timeout -r 1 [ 1269.413651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e network_timeout -r 1 [ 1269.440307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1269.468568] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1269.697564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1269.737589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1269.765404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1269.791780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1269.813977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.841846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.856107] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1269.858579] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.133@tcp [ 1269.871576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1269.895591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.910584] LNet: 65972:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.936834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1269.979144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.015787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.066499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.108797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.144848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1274.905505] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1275.267502] LNet: 66046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.283874] LNetError: 66046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.297749] LNet: Removed LNI 192.168.206.33@tcp [ 1275.313356] LNet: Removed LNI 192.168.206.33@tcp1 [ 1275.769820] Key type .llcrypt unregistered [ 1275.775682] Key type ._llcrypt unregistered [ 1292.461755] Key type ._llcrypt registered [ 1292.468670] Key type .llcrypt registered [ 1294.017569] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 05:05:41 (1788599141) [ 1294.433482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1294.456966] alg: No test for adler32 (adler32-zlib) [ 1295.423782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1295.503879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1295.558805] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1296.360951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1296.390770] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1297.334257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 1297.430703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1297.484770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1297.532567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1297.575304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1297.621988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1297.666418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.33@tcp [ 1297.689339] LNetError: 67146:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.33@tcp to recovery [ 1298.529551] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1298.532450] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1300.577633] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1301.240430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1304.673729] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1304.822896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1304.863590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1304.901467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1304.917881] LNet: 67253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.923802] LNetError: 67253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.929654] LNet: Removed LNI 192.168.206.33@tcp [ 1304.946691] LNet: Removed LNI 192.168.206.33@tcp1 [ 1304.971916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1305.055647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1305.095336] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1305.791793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1305.829726] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1306.621714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 1306.679248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.756970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1306.801658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1306.835961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1306.884330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1306.930483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1306.981845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.33@tcp [ 1307.015767] LNetError: 67525:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.33@tcp to recovery [ 1307.049207] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1309.465818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1313.184766] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1313.191408] LNet: Skipped 1 previous similar message [ 1317.471739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1317.520975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1317.546265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1324.331571] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 05:06:12 (1788599172) [ 1324.535970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1324.578831] LNet: 68233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.601306] LNetError: 68233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.607494] LNetError: 68233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1324.612723] LNet: Removed LNI 192.168.206.33@tcp [ 1324.625158] LNet: Removed LNI 192.168.206.33@tcp1 [ 1324.669624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.738649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1324.779577] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1325.558203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1325.576687] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1326.486591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 1326.575882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1326.631250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1326.679325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1326.718286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1326.756271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1326.793802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.33@tcp --health 0 [ 1326.912499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1327.776387] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.33@tcp [ 1327.777068] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1327.790672] LNet: Skipped 1 previous similar message [ 1341.366847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1341.483537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1341.524367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.556062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.33@tcp --health 500 [ 1341.653170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1347.304968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1347.382788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1347.416385] LNet: 68709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.423483] LNetError: 68709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.433793] LNetError: 68709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1347.441191] LNet: Removed LNI 192.168.206.33@tcp [ 1347.459182] LNet: Removed LNI 192.168.206.33@tcp1 [ 1347.475670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1347.522730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1347.565907] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1348.286283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1348.324687] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1349.223627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 1349.270744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.355278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1349.413411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1349.446284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1349.488915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1349.531133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1349.561859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.33@tcp --health 0 [ 1350.561586] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1350.570132] LNet: Skipped 3 previous similar messages [ 1350.879948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1361.163483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1361.191515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1361.221723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1367.589804] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 05:06:55 (1788599215) [ 1367.897488] LNet: 69754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.911888] LNetError: 69754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.917575] LNetError: 69754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1367.922575] LNet: Removed LNI 192.168.206.33@tcp [ 1367.931555] LNet: Removed LNI 192.168.206.33@tcp1 [ 1368.596772] Key type .llcrypt unregistered [ 1368.599169] Key type ._llcrypt unregistered [ 1373.856222] Key type ._llcrypt registered [ 1373.858794] Key type .llcrypt registered [ 1374.089621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1374.105244] alg: No test for adler32 (adler32-zlib) [ 1375.029290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1375.067076] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1382.547655] Lustre: DEBUG MARKER: Initial discovery [ 1382.604751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.133@tcp [ 1384.693476] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1384.748588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1384.796359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.133@tcp [ 1384.829248] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1384.849686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1384.892368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.133@tcp -r 1 -m GET -l 3 [ 1384.969384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.133@tcp -r 1 -m GET -e local_error [ 1385.010511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.133@tcp -r 1 -m PUT -l 6 [ 1385.319963] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1386.259471] Lustre: DEBUG MARKER: Force oleg633-client.virtnet to discover 192.168.206.133@tcp (in background) [ 1386.333925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.133@tcp [ 1387.746382] Lustre: DEBUG MARKER: Force oleg633-server to discover 192.168.206.33@tcp [ 1389.408501] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1390.052324] Lustre: DEBUG MARKER: Wait for 70506 [ 1391.693741] Lustre: DEBUG MARKER: Finished wait on 70506 [ 1391.793026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1391.831542] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1391.838345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1392.161711] LNet: 70966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.171454] LNetError: 70966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.183956] LNet: Removed LNI 192.168.206.33@tcp [ 1392.683504] Key type .llcrypt unregistered [ 1392.688168] Key type ._llcrypt unregistered [ 1407.996611] Key type ._llcrypt registered [ 1407.999806] Key type .llcrypt registered [ 1409.124898] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 05:07:37 (1788599257) [ 1409.666882] Key type .llcrypt unregistered [ 1409.668528] Key type ._llcrypt unregistered [ 1414.994545] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1415.172230] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1415.176064] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1415.185128] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1417.637610] Key type ._llcrypt registered [ 1417.642256] Key type .llcrypt registered [ 1417.854149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1417.872617] alg: No test for adler32 (adler32-zlib) [ 1418.807386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.858479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1418.881453] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1419.520804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1419.539692] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1428.100781] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 05:07:55 (1788599275) [ 1428.179318] LNet: 73036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.183923] LNetError: 73036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.188988] LNet: Removed LNI 192.168.206.33@tcp [ 1428.205924] LNet: Removed LNI 10.1.2.3@tcp [ 1428.812065] Key type .llcrypt unregistered [ 1428.814558] Key type ._llcrypt unregistered [ 1434.674852] Key type ._llcrypt registered [ 1434.679190] Key type .llcrypt registered [ 1434.853957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1434.869426] alg: No test for adler32 (adler32-zlib) [ 1435.780714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.815428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.834895] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1436.505464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1436.523288] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1437.480832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.33@tcp [ 1438.563268] LNetError: Unexpected error -22 connecting to 192.168.206.33@tcp at host 192.168.206.33:988 [ 1445.160854] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 05:08:12 (1788599292) [ 1445.279975] LNet: 74168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.286902] LNetError: 74168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1445.292377] LNet: Removed LNI 192.168.206.33@tcp [ 1445.303088] LNet: Removed LNI 10.1.2.3@tcp [ 1446.046587] Key type .llcrypt unregistered [ 1446.052697] Key type ._llcrypt unregistered [ 1453.467431] Key type ._llcrypt registered [ 1453.472694] Key type .llcrypt registered [ 1453.975529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1454.008185] alg: No test for adler32 (adler32-zlib) [ 1455.004332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.099166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1455.143885] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1456.346518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1456.401478] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 1457.785492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.33@tcp1 --nid 192.168.206.33@tcp2 [ 1463.106282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp1 192.168.206.33@tcp2 [ 1463.153406] LNetError: 75013:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.33@tcp1 to 192.168.206.33@tcp2 - Route table may be misconfigured [ 1463.206483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp2 192.168.206.33@tcp1 [ 1470.196550] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 05:08:37 (1788599317) [ 1470.419435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1470.471748] LNet: 75583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.478846] LNetError: 75583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.520715] LNet: Removed LNI 192.168.206.33@tcp1 [ 1471.524260] LNetError: 75583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.536402] LNet: Removed LNI 192.168.206.33@tcp2 [ 1471.586590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.668561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.710955] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1472.352717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1472.376683] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1473.263811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 1473.308455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.33@tcp [ 1473.372603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp1 192.168.206.33@tcp1 [ 1473.471390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.33@tcp -d 192.168.206.33@tcp -e network_timeout [ 1473.549786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -e network_timeout [ 1473.607713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -e network_timeout [ 1473.664831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -e network_timeout [ 1473.738287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1473.780343] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1473.818575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1473.959821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1474.081589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1474.720920] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1480.598420] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 05:08:48 (1788599328) [ 1480.687389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.712407] LNet: 76442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.719641] LNetError: 76442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.727462] LNet: Removed LNI 192.168.206.33@tcp [ 1481.771553] LNet: Removed LNI 192.168.206.33@tcp1 [ 1481.793102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.873498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1482.080166] LNet: 76484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.416426] Key type .llcrypt unregistered [ 1482.422529] Key type ._llcrypt unregistered [ 1492.865187] Key type ._llcrypt registered [ 1492.871979] Key type .llcrypt registered [ 1494.345122] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 05:09:01 (1788599341) [ 1494.680328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1494.700208] alg: No test for adler32 (adler32-zlib) [ 1495.583465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.826957] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1498.287924] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 05:09:06 (1788599346) [ 1498.492293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1498.520993] LNet: 77582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1498.551874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1498.602966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1498.623080] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1499.174425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1499.203915] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1499.941266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1499.980860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp1 [ 1500.019366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp1 [ 1505.936070] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 05:09:13 (1788599353) [ 1507.621465] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg633-server [ 1508.982769] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 05:09:17 (1788599357) [ 1510.702365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg633-server [ 1512.099920] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 05:09:20 (1788599360) [ 1513.751777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg633-server [ 1515.281605] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 05:09:23 (1788599363) [ 1516.710821] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg633-server [ 1518.254557] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 05:09:26 (1788599366) [ 1520.002945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg633-server [ 1521.479803] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 05:09:29 (1788599369) [ 1523.344114] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg633-server [ 1524.771729] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 05:09:32 (1788599372) [ 1526.431346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg633-server [ 1527.732568] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 05:09:35 (1788599375) [ 1529.384974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg633-server [ 1530.916424] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 05:09:38 (1788599378) [ 1532.543407] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg633-server [ 1533.975427] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 05:09:42 (1788599382) [ 1534.069935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.098673] LNet: 81454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.104123] LNetError: 81454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.109506] LNet: Removed LNI 192.168.206.33@tcp [ 1535.146809] LNet: Removed LNI 192.168.206.33@tcp1 [ 1535.188419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.251387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.283309] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1536.003331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1536.085911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1536.249336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1536.279356] LNet: 81606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1536.286698] LNetError: 81606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1536.290595] LNetError: 81606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1537.330695] LNet: Removed LNI 192.168.206.33@tcp [ 1537.354715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.401370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1537.416419] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1538.075630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1538.230563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.252613] LNetError: 81752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.312213] LNet: Removed LNI 192.168.206.33@tcp [ 1539.348733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.415430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.443979] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1540.320724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1540.417971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1540.631868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.654700] LNet: 81899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.668499] LNet: 81899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1540.675415] LNetError: 81899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1541.735173] LNet: Removed LNI 192.168.206.33@tcp [ 1541.758000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.815713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1541.843606] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1542.490435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1542.677413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.719546] LNet: Removed LNI 192.168.206.33@tcp [ 1543.753540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.809848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.824849] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1544.541891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1544.617249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1544.828119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1544.852905] LNet: 82192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.859582] LNet: 82192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1544.874976] LNetError: 82192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.886515] LNetError: 82192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1544.911502] LNet: Removed LNI 192.168.206.33@tcp [ 1544.937172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.005460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1545.044965] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1545.790186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1546.065447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.110913] LNet: Removed LNI 192.168.206.33@tcp [ 1547.147428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.232248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1547.278411] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1548.057275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1548.114276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1549.567569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.631991] LNet: Removed LNI 192.168.206.33@tcp [ 1550.673580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.766947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1550.814741] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1551.639383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1551.801974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.883656] LNet: Removed LNI 192.168.206.33@tcp [ 1552.913195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.976422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1552.994670] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1553.619780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1553.707409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1555.037541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.059251] LNet: 82795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.063188] LNet: 82795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1555.068517] LNetError: 82795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.075680] LNetError: 82795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1555.091432] LNet: Removed LNI 192.168.206.33@tcp [ 1555.120181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.174648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1555.198055] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1555.966721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1556.208366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.295574] LNet: Removed LNI 192.168.206.33@tcp [ 1557.335660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.386631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1557.414332] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1558.058125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1558.144403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1558.375537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.467404] LNet: Removed LNI 192.168.206.33@tcp [ 1559.494884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.585382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1559.611417] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1560.341122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1560.620823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.706720] LNet: Removed LNI 192.168.206.33@tcp [ 1561.777683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.855347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.880727] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1562.658359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1562.732966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.773247] LNet: Removed LNI 192.168.206.33@tcp [ 1562.790842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.856938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.905606] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1563.602809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1570.084945] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 05:10:18 (1788599418) [ 1570.183215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.237388] LNet: Removed LNI 192.168.206.33@tcp [ 1570.282350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.391995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1570.420882] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/180] [ 1571.729309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1571.749916] LNet: 84208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.758460] LNet: 84208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1571.766484] LNetError: 84208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1571.770665] LNetError: 84208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1571.783854] LNet: Removed LNI 192.168.206.33@tcp231 [ 1571.828763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.937418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1571.995549] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/180] [ 1572.008350] LNet: 84216:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1572.520121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1572.563965] LNet: Removed LNI 192.168.206.33@tcp231 [ 1572.587227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1572.635185] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/180] [ 1574.208782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.266538] LNet: Removed LNI 192.168.206.33@tcp231 [ 1574.293127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.409789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1574.480644] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/0] [ 1574.486281] LNet: 84368:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1575.196848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1575.231496] LNet: Removed LNI 192.168.206.33@tcp231 [ 1575.261776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1575.301236] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/0] [ 1576.427285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.454161] LNet: Removed LNI 192.168.206.33@tcp231 [ 1576.475185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.537292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1576.564932] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/60] [ 1576.571348] LNet: 84520:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.051217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1577.103161] LNet: Removed LNI 192.168.206.33@tcp231 [ 1577.131395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1577.166418] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/60] [ 1578.759769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.804301] LNet: Removed LNI 192.168.206.33@tcp231 [ 1578.836933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.916933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1578.947212] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/180] [ 1578.959706] LNet: 84671:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1579.553881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1579.595697] LNet: Removed LNI 192.168.206.33@tcp231 [ 1579.623977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1579.677842] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/180] [ 1587.568415] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 05:10:35 (1788599435) [ 1587.823559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.870502] LNet: Removed LNI 192.168.206.33@tcp231 [ 1587.898235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.971121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1588.002800] LNet: Added LNI 192.168.206.33@tcp232 [8/256/0/180] [ 1588.674968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1588.727886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1588.762746] LNet: Removed LNI 192.168.206.33@tcp232 [ 1588.778462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1588.821690] LNet: Added LNI 192.168.206.33@tcp232 [8/256/0/180] [ 1589.667797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1589.704869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1589.743732] LNet: Removed LNI 192.168.206.33@tcp232 [ 1589.765103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1589.787209] LNet: Added LNI 192.168.206.33@tcp232 [8/256/0/180] [ 1590.664417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1590.699475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1590.726165] LNet: Removed LNI 192.168.206.33@tcp232 [ 1590.742799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1590.776932] LNet: Added LNI 192.168.206.33@tcp232 [8/256/0/180] [ 1591.658207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1591.726460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1591.774085] LNet: Removed LNI 192.168.206.33@tcp232 [ 1591.811271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1598.696213] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 05:10:46 (1788599446) [ 1599.841266] Key type .llcrypt unregistered [ 1599.844122] Key type ._llcrypt unregistered [ 1606.407585] Key type ._llcrypt registered [ 1606.411135] Key type .llcrypt registered [ 1606.730143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1606.764574] alg: No test for adler32 (adler32-zlib) [ 1607.755433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1607.791576] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1615.347874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1623.237203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.278413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1623.304856] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1624.811940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_interrupt -r 2 [ 1624.855782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_interrupt -r 2 [ 1624.912706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_interrupt -r 2 [ 1624.955785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_interrupt -r 2 [ 1625.004337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_interrupt -r 2 [ 1625.059916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_interrupt -r 2 [ 1625.113878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_interrupt -r 2 [ 1625.172566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_interrupt -r 2 [ 1625.217608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1625.262512] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1625.285743] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1625.634137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1625.663251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1625.688915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1625.711352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1625.760272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1625.803989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1625.859905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1625.897375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1625.918993] LNet: 87171:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1625.932848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1625.961071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1625.990580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1626.016836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1626.040764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1626.071102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1626.715630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_dropped -r 2 [ 1626.746175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_dropped -r 2 [ 1626.776500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_dropped -r 2 [ 1626.804283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_dropped -r 2 [ 1626.851385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_dropped -r 2 [ 1626.909969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_dropped -r 2 [ 1626.938344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_dropped -r 2 [ 1626.978982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_dropped -r 2 [ 1627.019839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1627.046815] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1627.269752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1627.304890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1627.338292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1627.364324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1627.387094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1627.412125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1627.437855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1627.464860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1627.481881] LNet: 87291:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1627.495835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1627.518499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1627.543523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1627.567811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1627.597293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1627.625580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1628.021916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_aborted -r 2 [ 1628.060369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_aborted -r 2 [ 1628.089915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_aborted -r 2 [ 1628.137519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_aborted -r 2 [ 1628.183767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_aborted -r 2 [ 1628.226678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_aborted -r 2 [ 1628.254526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_aborted -r 2 [ 1628.291920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_aborted -r 2 [ 1628.340445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1628.361969] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1628.514645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1628.534121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1628.625720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1628.673683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1628.744830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1628.814094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1628.857727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1628.899698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1628.929749] LNet: 87411:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1628.958946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1628.999674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1629.034387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1629.059714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1629.088473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1629.131808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1629.617818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_no_route -r 2 [ 1629.702210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_no_route -r 2 [ 1629.780786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_no_route -r 2 [ 1629.827913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_no_route -r 2 [ 1629.867676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_no_route -r 2 [ 1629.905130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_no_route -r 2 [ 1629.946777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_no_route -r 2 [ 1629.993018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_no_route -r 2 [ 1630.042252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1630.302775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1630.341836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1630.395947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1630.426587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1630.446670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1630.480237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1630.520053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1630.562545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1630.610378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1630.646879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1630.705384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1630.756856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1630.797238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1630.818782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1631.243788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_timeout -r 2 [ 1631.265872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_timeout -r 2 [ 1631.291937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_timeout -r 2 [ 1631.323250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_timeout -r 2 [ 1631.350345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_timeout -r 2 [ 1631.378100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_timeout -r 2 [ 1631.408043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_timeout -r 2 [ 1631.433273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_timeout -r 2 [ 1631.463855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1631.486386] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1631.491102] LNet: Skipped 1 previous similar message [ 1631.745996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1631.786280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1631.806817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1631.832380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1631.861834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1631.885413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1631.919954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1631.957168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1631.971612] LNet: 87651:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1631.976676] LNet: 87651:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1631.994595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1632.026374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1632.082925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1632.120476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1632.158113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1632.209410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1632.882927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_dropped -r 2 [ 1632.911542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e remote_dropped -r 2 [ 1632.954610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_dropped -r 2 [ 1632.988079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e remote_dropped -r 2 [ 1633.029741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e remote_dropped -r 2 [ 1633.061981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e remote_dropped -r 2 [ 1633.094783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e remote_dropped -r 2 [ 1633.140663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e remote_dropped -r 2 [ 1633.182991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1633.206645] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.133@tcp [ 1633.403817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1633.436276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1633.463429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1633.484683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1633.507867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1633.537315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1633.576302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1633.631975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1633.655571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1633.685784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1633.716625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1633.743639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1633.769556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1633.805275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1640.097545] LNet: 87846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.114903] LNetError: 87846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.126546] LNet: Removed LNI 192.168.206.33@tcp [ 1640.139208] LNet: Removed LNI 192.168.206.33@tcp1 [ 1640.494573] Key type .llcrypt unregistered [ 1640.496395] Key type ._llcrypt unregistered [ 1658.348791] Key type ._llcrypt registered [ 1658.350646] Key type .llcrypt registered [ 1659.470628] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 05:11:47 (1788599507) [ 1659.893195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1659.915886] alg: No test for adler32 (adler32-zlib) [ 1660.876927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.960807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.983474] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1661.704194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1661.748161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.33@tcp [ 1661.770689] LNetError: 88825:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.33@tcp to recovery [ 1661.871151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.33@tcp [ 1661.986957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1669.137162] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 05:11:56 (1788599516) [ 1669.213555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.249603] LNet: 89409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.257726] LNetError: 89409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1669.281333] LNet: Removed LNI 192.168.206.33@tcp [ 1669.310530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.373277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.396830] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1669.933978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1669.959257] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1670.813758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.33@tcp [ 1670.834692] LNetError: 89680:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.206.33@tcp to recovery [ 1671.395247] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1671.401819] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1672.085343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.33@tcp [ 1681.120640] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 05:12:08 (1788599528) [ 1683.328578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg633-server [ 1685.903903] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 05:12:13 (1788599533) [ 1687.999099] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg633-server [ 1690.666421] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 05:12:18 (1788599538) [ 1693.481237] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg633-server [ 1695.880489] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 05:12:23 (1788599543) [ 1696.156857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.200877] LNet: 91320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.206987] LNetError: 91320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1696.214111] LNet: Removed LNI 192.168.206.33@tcp [ 1696.239534] LNet: Removed LNI 192.168.206.33@tcp1 [ 1696.298482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.387802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.419934] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1697.506619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1697.603868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1697.687582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1697.793398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1697.841937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1697.894873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1705.344585] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 05:12:32 (1788599552) [ 1705.427254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.453213] LNet: 92040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.461714] LNetError: 92040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.473564] LNetError: 92040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1705.496924] LNet: Removed LNI 192.168.206.33@tcp [ 1705.533779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.615035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.642316] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1712.769798] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 05:12:40 (1788599560) [ 1712.995687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.037683] LNet: 92750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.051172] LNetError: 92750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.075395] LNet: Removed LNI 192.168.206.33@tcp [ 1713.097992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.231757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.273223] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1721.035358] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 05:12:48 (1788599568) [ 1722.749913] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1724.548948] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 05:12:52 (1788599572) [ 1725.125361] LNet: 93785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.131730] LNetError: 93785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.145184] LNet: Removed LNI 192.168.206.33@tcp [ 1726.037129] Key type .llcrypt unregistered [ 1726.040136] Key type ._llcrypt unregistered [ 1732.607064] Key type ._llcrypt registered [ 1732.612516] Key type .llcrypt registered [ 1732.919415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1732.954615] alg: No test for adler32 (adler32-zlib) [ 1733.955623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1733.985490] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1740.708692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1745.208538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.133@tcp [ 1745.507394] LNet: 94202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.516129] LNetError: 94202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.536489] LNet: Removed LNI 192.168.206.33@tcp [ 1745.945408] Key type .llcrypt unregistered [ 1745.948455] Key type ._llcrypt unregistered [ 1757.487123] Key type ._llcrypt registered [ 1757.490534] Key type .llcrypt registered [ 1758.707241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1760.213674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1761.990423] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 05:13:29 (1788599609) [ 1762.557185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1762.569781] alg: No test for adler32 (adler32-zlib) [ 1763.573891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.687868] LNet: 95289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.526308] Key type .llcrypt unregistered [ 1764.528693] Key type ._llcrypt unregistered [ 1771.217727] Key type ._llcrypt registered [ 1771.219362] Key type .llcrypt registered [ 1771.447864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1771.465946] alg: No test for adler32 (adler32-zlib) [ 1772.521151] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1772.527142] LNet: 95545:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1772.538974] LNet: 95545:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1772.574565] LNetError: 95557:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 1772.601064] LNetError: 95557:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.206.34@tcp has gone from up to down [ 1773.093319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.110906] LNet: 95603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.118940] LNetError: 95603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.142669] LNet: Removed LNI 192.168.206.33@tcp [ 1773.215629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.292290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1773.332248] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1773.338499] LNet: 95610:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1773.351226] LNet: 95610:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1773.364404] LNetError: 95610:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.206.34@tcp is a gateway. Can not delete it [ 1773.381066] LNetError: 95610:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.206.34@tcp is a router. Can not be deleted [ 1773.392289] LNetError: 95607:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.206.38@tcp is being used as a gateway but routing feature is not turned on [ 1773.400953] LNetError: 95607:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 8 previous similar messages [ 1773.411970] LNetError: 95607:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.206.38@tcp has gone from up to down [ 1773.421290] LNetError: 95607:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1773.439128] LNet: 95610:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1780.175934] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 05:13:47 (1788599627) [ 1782.151092] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg633-server [ 1783.925128] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 05:13:51 (1788599631) [ 1785.726662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg633-server [ 1787.487220] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 05:13:55 (1788599635) [ 1789.039414] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 1790.976193] Key type lgssc registered [ 1792.074365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1810.574403] Key type lgssc unregistered [ 1810.909503] LNet: 98031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.918380] LNetError: 98031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.970735] LNet: Removed LNI 192.168.206.33@tcp [ 1811.723558] Key type .llcrypt unregistered [ 1811.730927] Key type ._llcrypt unregistered [ 1829.769281] Key type ._llcrypt registered [ 1829.772414] Key type .llcrypt registered [ 1830.819169] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 05:14:38 (1788599678) [ 1831.538783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1831.554839] alg: No test for adler32 (adler32-zlib) [ 1832.397785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.451745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.523082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.562458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.604604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1839.503197] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 05:14:47 (1788599687) [ 1839.786326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1839.858749] LNet: 99437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.923534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.029787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1840.060945] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1840.995098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.041739] LNet: 99585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.053837] LNetError: 99585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.076842] LNet: Removed LNI 192.168.206.33@tcp [ 1841.113991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.181627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1841.213575] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1841.624666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.646299] LNetError: 99613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.660134] LNet: Removed LNI 192.168.206.33@tcp [ 1841.683391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.744318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1841.777176] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1849.339692] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 05:14:56 (1788599696) [ 1849.484496] LNet: 100194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1849.498831] LNet: 100194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1849.511110] LNetError: 100194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1849.537419] LNet: Removed LNI 192.168.206.33@tcp [ 1850.417555] Key type .llcrypt unregistered [ 1850.422860] Key type ._llcrypt unregistered [ 1858.201671] Key type ._llcrypt registered [ 1858.208134] Key type .llcrypt registered [ 1858.469849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1858.491854] alg: No test for adler32 (adler32-zlib) [ 1859.379410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.481439] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1859.484798] LNet: *** cfs_fail_loc=e003, val=100*** [ 1860.537060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1860.560040] LNet: *** cfs_fail_loc=e003, val=100*** [ 1860.564560] LNet: Skipped 2 previous similar messages [ 1860.571672] Ping reply received with mapping: [ 1860.577887] found metadata reply NI[192.168.206.33@tcp] [ 1860.580883] 21 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1860.587754] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1860.590959] 5a 5a 5a 5a 5a 5a 5a 5a [ 1860.874849] LNet: 100601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1860.885568] LNetError: 100601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1861.930691] LNet: Removed LNI 192.168.206.33@tcp [ 1862.527681] Key type .llcrypt unregistered [ 1862.529841] Key type ._llcrypt unregistered [ 1875.731163] Key type ._llcrypt registered [ 1875.732849] Key type .llcrypt registered [ 1877.495812] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 05:15:24 (1788599724) [ 1878.064401] Key type .llcrypt unregistered [ 1878.066734] Key type ._llcrypt unregistered [ 1884.599699] Key type ._llcrypt registered [ 1884.609719] Key type .llcrypt registered [ 1884.701066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1884.711660] alg: No test for adler32 (adler32-zlib) [ 1885.723881] LNetError: 101646:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1886.145431] Key type .llcrypt unregistered [ 1886.148432] Key type ._llcrypt unregistered [ 1891.514565] Key type ._llcrypt registered [ 1891.519114] Key type .llcrypt registered [ 1892.645877] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 05:15:40 (1788599740) [ 1893.122400] Key type .llcrypt unregistered [ 1893.123872] Key type ._llcrypt unregistered [ 1898.610330] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1898.614503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1901.114996] Key type ._llcrypt registered [ 1901.117681] Key type .llcrypt registered [ 1901.362496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1901.400416] alg: No test for adler32 (adler32-zlib) [ 1902.372477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.476526] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1911.734919] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 05:15:59 (1788599759) [ 1911.874308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.927660] LNet: 103420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.932978] LNetError: 103420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1911.954644] LNet: Removed LNI 10.1.2.3@tcp [ 1911.991079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.674605] Key type .llcrypt unregistered [ 1912.679551] Key type ._llcrypt unregistered [ 1918.953898] Key type ._llcrypt registered [ 1918.957401] Key type .llcrypt registered [ 1919.095657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1919.108434] alg: No test for adler32 (adler32-zlib) [ 1920.111218] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1920.208527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1920.348666] LNet: 103714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.354949] LNetError: 103714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.366403] LNet: Removed LNI 192.168.206.33@tcp [ 1920.980728] Key type .llcrypt unregistered [ 1920.987217] Key type ._llcrypt unregistered [ 1931.013847] Key type ._llcrypt registered [ 1931.016512] Key type .llcrypt registered [ 1932.076968] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 05:16:20 (1788599780) [ 1932.555967] Key type .llcrypt unregistered [ 1932.558385] Key type ._llcrypt unregistered [ 1938.960885] Key type ._llcrypt registered [ 1938.963518] Key type .llcrypt registered [ 1939.191674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1939.215373] alg: No test for adler32 (adler32-zlib) [ 1945.289977] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 05:16:33 (1788599793) [ 1945.439831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1945.507462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.586721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1945.649569] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1946.374885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1946.419315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1946.449383] LNet: Removed LNI 192.168.206.33@tcp [ 1946.688846] LNet: 105457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.044434] Key type .llcrypt unregistered [ 1947.049624] Key type ._llcrypt unregistered [ 1955.836314] Key type ._llcrypt registered [ 1955.839083] Key type .llcrypt registered [ 1956.832468] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 05:16:44 (1788599804) [ 1957.116377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1957.133865] alg: No test for adler32 (adler32-zlib) [ 1958.040109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.088255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1958.112958] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1958.700373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1958.733860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.33@tcp [ 1958.749161] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1958.754053] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1958.756916] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1958, 0, 0 [ 1958.761352] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1958.769436] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1958.775205] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1958.778472] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1958, 0, 0 [ 1958.784175] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1958.786354] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1958.789427] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1958.792135] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1958, 0, 2008 [ 1958.795445] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1958.797185] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1958.800351] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1958.803357] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1958, 0, 2008 [ 1958.807279] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1963.906115] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 05:16:51 (1788599811) [ 1964.365513] LNet: 106997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.376212] LNetError: 106997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.426130] LNet: Removed LNI 192.168.206.33@tcp [ 1965.931539] Key type .llcrypt unregistered [ 1965.933165] Key type ._llcrypt unregistered [ 1971.631697] Key type ._llcrypt registered [ 1971.636886] Key type .llcrypt registered [ 1971.834342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1971.848442] alg: No test for adler32 (adler32-zlib) [ 1972.775869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1972.801647] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1980.564588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1989.368194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.415180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1989.443347] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1990.351262] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1990.534731] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1990.544103] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1990.555699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1992.947440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1992.965713] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2002.749297] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 05:17:30 (1788599850) [ 2002.890229] LNet: 108500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.897256] LNetError: 108500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.912123] LNet: Removed LNI 192.168.206.33@tcp [ 2003.936205] LNet: Removed LNI 192.168.206.33@tcp1 [ 2003.939105] LNetError: 108500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2003.944697] LNetError: 108500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2003.966744] LNet: Removed LNI 10.1.2.3@tcp99 [ 2004.532898] Key type .llcrypt unregistered [ 2004.540878] Key type ._llcrypt unregistered [ 2010.018688] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2010.024240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2012.472944] Key type ._llcrypt registered [ 2012.474670] Key type .llcrypt registered [ 2012.650676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2012.663174] alg: No test for adler32 (adler32-zlib) [ 2013.520494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2013.586397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2013.614213] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2014.366810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2014.381660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2016.222794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2016.295681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2016.332132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2016.354669] LNetError: 109248:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2016.385664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2016.451683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2016.871170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2016.920748] LNet: 109269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.931821] LNetError: 109269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.938334] LNet: Removed LNI 192.168.206.33@tcp [ 2016.950168] LNet: Removed LNI 10.1.2.3@tcp [ 2017.565873] Key type .llcrypt unregistered [ 2017.575167] Key type ._llcrypt unregistered [ 2028.358656] Key type ._llcrypt registered [ 2028.360880] Key type .llcrypt registered [ 2029.771737] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 05:17:57 (1788599877) [ 2030.240208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2030.257599] alg: No test for adler32 (adler32-zlib) [ 2031.164543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2031.220105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2031.249496] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2031.896744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2032.001908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg633-client@tcp [ 2038.262084] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 05:18:06 (1788599886) [ 2038.532427] LNet: 110804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.556624] LNetError: 110804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.591023] LNet: Removed LNI 192.168.206.33@tcp [ 2039.260441] Key type .llcrypt unregistered [ 2039.263063] Key type ._llcrypt unregistered [ 2045.689397] Key type ._llcrypt registered [ 2045.695332] Key type .llcrypt registered [ 2045.874640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.896841] alg: No test for adler32 (adler32-zlib) [ 2046.813974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2046.872328] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2046.917074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2053.093902] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 05:18:21 (1788599901) [ 2054.649908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2056.376633] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 05:18:24 (1788599904) [ 2057.728453] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2059.246967] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 05:18:27 (1788599907) [ 2059.439903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.462846] LNet: 112223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.468634] LNetError: 112223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.482623] LNet: Removed LNI 192.168.206.33@tcp [ 2059.513930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.584616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.617407] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2060.283578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2060.319675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2066.305133] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 05:18:34 (1788599914) [ 2066.415557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2066.443796] LNet: 112929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2066.450584] LNetError: 112929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2067.496663] LNet: Removed LNI 192.168.206.33@tcp [ 2067.537849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.624302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2067.656251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2067.990295] LNet: 112968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2068.298448] Key type .llcrypt unregistered [ 2068.301819] Key type ._llcrypt unregistered [ 2078.203503] Key type ._llcrypt registered [ 2078.205896] Key type .llcrypt registered [ 2079.523412] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 05:18:47 (1788599927) [ 2079.850250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2079.862119] alg: No test for adler32 (adler32-zlib) [ 2080.835454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.914911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2080.950270] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2081.662636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2081.764908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2087.706494] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 05:18:55 (1788599935) [ 2087.978394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.007653] LNet: 114487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.018585] LNetError: 114487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2088.038989] LNet: Removed LNI 192.168.206.33@tcp [ 2088.074909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.130842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2088.179453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2094.054345] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 05:19:02 (1788599942) [ 2094.409215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2094.434618] LNet: 115067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.466269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.596871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2095.069336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2095.109366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2095.170440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2095.442680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2095.467269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2095.517517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2095.534937] LNet: 115101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2095.550951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.607512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2095.997576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2096.030313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2096.080607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2096.378909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2096.406553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2102.684845] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 05:19:10 (1788599950) [ 2103.148160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2103.194531] LNet: 115701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.222673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.264458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2103.279319] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2104.094678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2104.113627] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2104.858748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2104.904174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2105.069412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.088550] LNetError: 115980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.093544] LNet: Removed LNI 192.168.206.33@tcp [ 2105.101626] LNet: Removed LNI 192.168.206.33@tcp2 [ 2105.143937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.219484] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2105.226075] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2105.231681] LNet: 115986:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2105.765318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2105.979591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.040126] LNet: Removed LNI 192.168.206.33@tcp [ 2107.046352] LNetError: 116023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.056422] LNetError: 116023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2107.070357] LNet: Removed LNI 192.168.206.33@tcp2 [ 2107.128432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.232179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2107.276220] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2107.925600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2107.962936] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2108.989283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2109.022174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2109.064742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2109.226519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2109.242464] LNet: 116308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2109.246221] LNet: 116308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2109.251672] LNetError: 116308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.258580] LNet: Removed LNI 192.168.206.33@tcp [ 2110.312536] LNet: Removed LNI 192.168.206.33@tcp2 [ 2110.342270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2110.428918] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2110.441650] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2110.459484] LNet: 116314:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2111.063847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2111.365625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.403326] LNet: Removed LNI 192.168.206.33@tcp [ 2112.422747] LNet: Removed LNI 192.168.206.33@tcp2 [ 2112.463305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.562497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2112.595438] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2113.355547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2113.371920] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2114.123993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2114.208558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2114.554571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2114.581359] LNetError: 116631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.586684] LNetError: 116631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2115.616144] LNet: Removed LNI 192.168.206.33@tcp [ 2115.631928] LNet: Removed LNI 192.168.206.33@tcp2 [ 2115.667522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.724240] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2115.733196] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2115.741857] LNet: 116637:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2116.409610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2116.625738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.664879] LNet: Removed LNI 192.168.206.33@tcp [ 2117.684851] LNet: Removed LNI 192.168.206.33@tcp2 [ 2117.736751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.823523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2117.856205] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2118.612950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2118.681881] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2119.709712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2119.768918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2119.850841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2120.137068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.165993] LNet: 116958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.175698] LNet: 116958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2121.248285] LNet: Removed LNI 192.168.206.33@tcp [ 2122.279301] LNet: Removed LNI 192.168.206.33@tcp2 [ 2122.326710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.395407] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2122.401610] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2122.406602] LNet: 116965:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2122.985260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2123.239310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.259527] LNetError: 117002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.264502] LNetError: 117002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2123.274888] LNet: Removed LNI 192.168.206.33@tcp [ 2124.325962] LNet: Removed LNI 192.168.206.33@tcp2 [ 2124.381245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.470824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2124.503788] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2125.305389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2125.323906] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2126.106958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2126.146340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2126.207470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2126.618880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.648632] LNet: Removed LNI 192.168.206.33@tcp [ 2127.662778] LNet: Removed LNI 192.168.206.33@tcp2 [ 2127.722457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.831740] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2127.838481] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2127.847501] LNet: 117305:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2128.557332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2128.896285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.952170] LNet: Removed LNI 192.168.206.33@tcp [ 2129.985602] LNet: Removed LNI 192.168.206.33@tcp2 [ 2130.037359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.111508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2130.134789] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2130.881469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2130.917942] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2131.871074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2131.906354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2131.931068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2131.981410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2132.303856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.344243] LNet: Removed LNI 192.168.206.33@tcp [ 2134.374653] LNet: Removed LNI 192.168.206.33@tcp2 [ 2134.392406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.464630] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2134.470242] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2135.070352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2141.753247] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2143.727994] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 05:19:51 (1788599991) [ 2143.813467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.845472] LNet: 118417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.848252] LNet: 118417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2143.855753] LNetError: 118417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2143.866336] LNetError: 118417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2143.874834] LNet: Removed LNI 192.168.206.33@tcp [ 2144.935620] LNet: Removed LNI 192.168.206.33@tcp2 [ 2144.984581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.283055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2145.314629] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2145.979607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2146.000118] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2147.016657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.073843] LNet: Removed LNI 192.168.206.33@tcp [ 2147.100288] LNet: Removed LNI 10.1.2.3@tcp [ 2147.137556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.241573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2147.283517] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2147.289406] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2147.785199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2147.816390] LNet: Removed LNI 192.168.206.33@tcp [ 2147.830859] LNet: Removed LNI 10.1.2.3@tcp [ 2154.131152] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 05:20:02 (1788600002) [ 2154.266272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.306995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.429469] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2154.542633] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2154.547675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2154.555861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2156.949943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2156.990486] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2157.852643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2157.869733] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2160.186940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2160.261194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2160.342306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.33@tcp -m PUT -e local_timeout -r 1 [ 2160.440723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2160.501202] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2160.506720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2160.725966] LNetError: 119362:0:(lib-move.c:4969:lnet_parse()) 192.168.206.33@tcp, src 192.168.206.33@tcp: Dropping PUT (error -108 looking up sender) [ 2162.784215] LNet: Removed LNI 192.168.206.33@tcp [ 2162.796720] LNet: Removed LNI 10.1.2.3@tcp [ 2163.133542] Key type .llcrypt unregistered [ 2163.136092] Key type ._llcrypt unregistered [ 2174.543846] Key type ._llcrypt registered [ 2174.548474] Key type .llcrypt registered [ 2175.722067] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 05:20:23 (1788600023) [ 2176.104248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2176.122526] alg: No test for adler32 (adler32-zlib) [ 2177.044940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.498765] LNet: 120673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2177.931433] Key type .llcrypt unregistered [ 2177.939739] Key type ._llcrypt unregistered [ 2183.424148] Key type ._llcrypt registered [ 2183.426786] Key type .llcrypt registered [ 2183.660879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2183.677996] alg: No test for adler32 (adler32-zlib) [ 2184.603361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2184.624532] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2194.132261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 2194.574133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2195.622824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.133@tcp [ 2195.873880] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.133@tcp [ 2196.183208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2196.252426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2196.295719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2196.341851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2196.393852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2196.433955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2196.494109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2196.553735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2196.594898] LNet: 121124:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2196.630731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2196.686981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2196.746540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2196.805572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2196.848453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2196.886282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2197.754734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2197.983873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2198.018227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2198.050240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2198.079235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2198.105995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2198.140753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2198.194585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2198.261742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2198.294391] LNet: 121241:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2198.322419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2198.363702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2198.417609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2198.467464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2198.496983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2198.530341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2199.058438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.133@tcp [ 2199.333206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2199.371875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2199.405739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2199.457172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2199.532915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2199.591725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2199.633784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2199.684889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2199.723575] LNet: 121358:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2199.750653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2199.792127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2199.844490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2199.910735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2199.960551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2199.994122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2200.628554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2200.984422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2201.019365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2201.069140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2201.105524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2201.156549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2201.204837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2201.266618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2201.317404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2201.357325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2201.405961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2201.479899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2201.539048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2201.621715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2201.716706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2202.084945] LNet: 121522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.091515] LNetError: 121522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.114104] LNet: Removed LNI 192.168.206.33@tcp [ 2202.682058] Key type .llcrypt unregistered [ 2202.695309] Key type ._llcrypt unregistered [ 2219.903376] Key type ._llcrypt registered [ 2219.916605] Key type .llcrypt registered [ 2221.221461] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 05:21:08 (1788600068) [ 2221.571956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2221.585090] alg: No test for adler32 (adler32-zlib) [ 2222.461226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.823646] LNet: 122425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.456361] Key type .llcrypt unregistered [ 2223.461285] Key type ._llcrypt unregistered [ 2229.118908] Key type ._llcrypt registered [ 2229.120468] Key type .llcrypt registered [ 2229.438706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2229.464267] alg: No test for adler32 (adler32-zlib) [ 2230.341978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2230.370540] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2236.664809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 2243.513802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.591675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2243.624991] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 2244.562807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2245.074568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.133@tcp [ 2245.302667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2245.349138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2245.375357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2245.412887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2245.451938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2245.483895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2245.515854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2245.547246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2245.587333] LNet: 123077:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2245.617631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2245.654935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2245.688841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2245.721822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2245.728387] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.133@tcp [ 2245.756343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2245.778950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2246.323341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2246.562231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2246.588146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2246.634498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2246.662603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2246.711280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2246.744923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2246.797996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2246.831498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2246.852489] LNet: 123194:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2246.866523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2246.907788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2246.938483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2246.967507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2246.999901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2247.039723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2247.689761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.133@tcp [ 2247.938337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2247.965607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2248.020545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2248.058300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2248.100431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2248.147082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2248.188468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2248.216510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2248.237752] LNet: 123312:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2248.264058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2248.303706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2248.348529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2248.380784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2248.418020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2248.463133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2249.023733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2249.225314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2249.258319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2249.286845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2249.316130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2249.345549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2249.391588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2249.427549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2249.464313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2249.495101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2249.517834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2249.540991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2249.565649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2249.598709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2249.633422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2253.510645] LNet: 123510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.515966] LNetError: 123510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.524902] LNet: Removed LNI 192.168.206.33@tcp [ 2253.531769] LNet: Removed LNI 192.168.206.33@tcp1 [ 2253.859902] Key type .llcrypt unregistered [ 2253.866887] Key type ._llcrypt unregistered [ 2268.236841] Key type ._llcrypt registered [ 2268.245531] Key type .llcrypt registered [ 2269.587194] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 05:21:57 (1788600117) [ 2276.161540] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 05:22:03 (1788600123) [ 2283.108791] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 05:22:11 (1788600131) [ 2289.306627] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 05:22:17 (1788600137) [ 2294.819462] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 05:22:22 (1788600142) [ 2301.336697] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 05:22:29 (1788600149) [ 2308.605363] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 05:22:36 (1788600156) [ 2315.074938] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 05:22:43 (1788600163) [ 2322.174450] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 05:22:50 (1788600170) [ 2328.541247] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 05:22:56 (1788600176) [ 2335.971905] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 05:23:03 (1788600183) [ 2343.368341] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 05:23:11 (1788600191) [ 2350.126826] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 05:23:17 (1788600197) [ 2356.607327] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 05:23:24 (1788600204) [ 2363.519885] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 05:23:31 (1788600211) [ 2370.360602] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 05:23:38 (1788600218) [ 2377.691710] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 05:23:45 (1788600225) [ 2385.409968] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 05:23:53 (1788600233) [ 2385.960433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2385.971521] alg: No test for adler32 (adler32-zlib) [ 2386.846971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2386.958636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2386.988615] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2388.246364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2388.268503] LNet: 134218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.277492] LNetError: 134218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2388.297881] LNet: Removed LNI 192.168.206.33@tcp [ 2388.320955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.412715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2388.436208] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2395.428693] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 05:24:03 (1788600243) [ 2402.177798] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 05:24:09 (1788600249) [ 2409.660337] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 05:24:17 (1788600257) [ 2418.555859] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 05:24:25 (1788600265) [ 2427.038638] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 05:24:34 (1788600274) [ 2433.965634] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 05:24:42 (1788600282) [ 2440.705944] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 05:24:48 (1788600288) [ 2446.800496] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 05:24:54 (1788600294) [ 2452.394100] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 05:25:00 (1788600300) [ 2458.499219] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 05:25:06 (1788600306) [ 2464.750882] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 05:25:12 (1788600312) [ 2470.786273] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 05:25:19 (1788600319) [ 2475.611290] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2251 sec ========== 05:25:23 (1788600323) [ 2475.891937] LNet: 141681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2475.900063] LNetError: 141681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2475.910709] LNet: Removed LNI 192.168.206.33@tcp [ 2476.556528] Key type .llcrypt unregistered [ 2476.558110] Key type ._llcrypt unregistered [ 2486.451926] Key type ._llcrypt registered [ 2486.453667] Key type .llcrypt registered [ 2486.807795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2486.830080] alg: No test for adler32 (adler32-zlib) [ 2487.936747] Lustre: Lustre: Build Version: 2.17.51_74_g53213db [ 2488.147125] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2489.792204] Key type lgssc registered [ 2490.785922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2556.166191] Lustre: Mounted lustre-client [ 2561.139686] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2574.241437] LustreError: 144139:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8cd082ad4800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2574.325597] Lustre: Unmounted lustre-client [ 2591.801822] Key type lgssc unregistered [ 2592.080556] LNet: 144618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2592.088620] LNetError: 144618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2592.114846] LNet: Removed LNI 192.168.206.33@tcp [ 2592.839283] Key type .llcrypt unregistered [ 2592.840962] Key type ._llcrypt unregistered