[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 462050945 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2878904K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.003247] x2apic enabled [ 0.004012] Switched APIC routing to physical x2apic. [ 0.005021] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.010165] LSM: Security Framework initializing [ 0.012010] Yama: becoming mindful. [ 0.013026] SELinux: Initializing. [ 0.014066] *** VALIDATE selinux *** [ 0.022674] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027083] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028186] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030156] *** VALIDATE tmpfs *** [ 0.032449] *** VALIDATE proc *** [ 0.034055] *** VALIDATE cgroup *** [ 0.035018] *** VALIDATE cgroup2 *** [ 0.037198] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038192] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041044] Spectre V2 : User space: Vulnerable [ 0.042017] Speculative Store Bypass: Vulnerable [ 0.045334] debug: unmapping init [mem 0xffffffff8ba59000-0xffffffff8ba60fff] [ 0.047956] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048788] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049031] ... version: 2 [ 0.050019] ... bit width: 48 [ 0.051017] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053019] ... max period: 00007fffffffffff [ 0.054023] ... fixed-purpose events: 3 [ 0.055019] ... event mask: 000000070000000f [ 0.056347] rcu: Hierarchical SRCU implementation. [ 0.058667] smp: Bringing up secondary CPUs ... [ 0.059698] x86: Booting SMP configuration: [ 0.060033] .... node #0, CPUs: #1 #2 #3 [ 0.063469] smp: Brought up 1 node, 4 CPUs [ 0.065022] smpboot: Max logical packages: 1 [ 0.066030] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.234353] node 0 deferred pages initialised in 166ms [ 0.239415] devtmpfs: initialized [ 0.240515] x86/mm: Memory block size: 128MB [ 0.243278] gcov: version magic: 0x41383552 [ 0.245743] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.251265] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256544] pinctrl core: initialized pinctrl subsystem [ 0.260560] [ 0.261012] ************************************************************* [ 0.265016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.268018] ** ** [ 0.272025] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.275023] ** ** [ 0.278019] ** This means that this kernel is built to expose internal ** [ 0.281094] ** IOMMU data structures, which may compromise security on ** [ 0.284019] ** your system. ** [ 0.287020] ** ** [ 0.290020] ** If you see this message and you are not debugging the ** [ 0.292019] ** kernel, report this immediately to your vendor! ** [ 0.295105] ** ** [ 0.298019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.300016] ************************************************************* [ 0.304328] NET: Registered protocol family 16 [ 0.306586] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.309095] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.312114] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.316135] cpuidle: using governor menu [ 0.317720] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.318678] PCI: Using configuration type 1 for base access [ 0.320153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.332182] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.333057] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.335186] cryptd: max_cpu_qlen set to 1000 [ 0.338012] ACPI: Added _OSI(Module Device) [ 0.338949] ACPI: Added _OSI(Processor Device) [ 0.339013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.340011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.343171] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.350691] ACPI: Interpreter enabled [ 0.352084] ACPI: PM: (supports S0 S3 S4 S5) [ 0.354017] ACPI: Using IOAPIC for interrupt routing [ 0.356174] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.359481] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.370255] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.373114] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.376027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.380106] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.385726] acpiphp: Slot [2] registered [ 0.387172] acpiphp: Slot [5] registered [ 0.389196] acpiphp: Slot [6] registered [ 0.391183] acpiphp: Slot [3] registered [ 0.393161] acpiphp: Slot [4] registered [ 0.394133] acpiphp: Slot [7] registered [ 0.396150] acpiphp: Slot [8] registered [ 0.398122] acpiphp: Slot [9] registered [ 0.399076] acpiphp: Slot [10] registered [ 0.400078] acpiphp: Slot [11] registered [ 0.401170] acpiphp: Slot [12] registered [ 0.403182] acpiphp: Slot [13] registered [ 0.405128] acpiphp: Slot [14] registered [ 0.407130] acpiphp: Slot [15] registered [ 0.408121] acpiphp: Slot [16] registered [ 0.410130] acpiphp: Slot [17] registered [ 0.412129] acpiphp: Slot [18] registered [ 0.414142] acpiphp: Slot [19] registered [ 0.415127] acpiphp: Slot [20] registered [ 0.417131] acpiphp: Slot [21] registered [ 0.419121] acpiphp: Slot [22] registered [ 0.420103] acpiphp: Slot [23] registered [ 0.422217] acpiphp: Slot [24] registered [ 0.424139] acpiphp: Slot [25] registered [ 0.426181] acpiphp: Slot [26] registered [ 0.427150] acpiphp: Slot [27] registered [ 0.429129] acpiphp: Slot [28] registered [ 0.431145] acpiphp: Slot [29] registered [ 0.433131] acpiphp: Slot [30] registered [ 0.434130] acpiphp: Slot [31] registered [ 0.436102] PCI host bridge to bus 0000:00 [ 0.437025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.440036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.443078] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.445057] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.448043] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.452040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.454251] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.457260] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.461428] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.468548] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.473099] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.477027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.479025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.482025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.485664] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.488911] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.491048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.494936] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.499015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.508718] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.513019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.521542] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.528021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.536030] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.551025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.562149] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.569069] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.574111] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.589030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.601000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.605639] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.608422] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.611439] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.614270] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.619183] iommu: Default domain type: Passthrough [ 0.621406] SCSI subsystem initialized [ 0.623159] ACPI: bus type USB registered [ 0.624128] usbcore: registered new interface driver usbfs [ 0.626069] usbcore: registered new interface driver hub [ 0.628078] usbcore: registered new device driver usb [ 0.629147] pps_core: LinuxPPS API ver. 1 registered [ 0.641020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.643082] PTP clock support registered [ 0.645128] EDAC MC: Ver: 3.0.0 [ 0.647338] PCI: Using ACPI for IRQ routing [ 0.648606] NetLabel: Initializing [ 0.649000] NetLabel: domain hash size = 128 [ 0.651128] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.654225] NetLabel: unlabeled traffic allowed by default [ 0.657166] vgaarb: loaded [ 0.658410] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.660014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.667886] clocksource: Switched to clocksource kvm-clock [ 0.779628] VFS: Disk quotas dquot_6.6.0 [ 0.781159] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.783819] *** VALIDATE ramfs *** [ 0.785143] *** VALIDATE hugetlbfs *** [ 0.786728] pnp: PnP ACPI init [ 0.789159] pnp: PnP ACPI: found 6 devices [ 0.807866] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.811026] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.813201] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.815309] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.817719] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.820236] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.823298] NET: Registered protocol family 2 [ 0.825777] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.830627] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.834296] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.841085] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.844327] TCP: Hash tables configured (established 65536 bind 65536) [ 0.847234] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.850561] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853619] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.856774] NET: Registered protocol family 1 [ 0.859771] RPC: Registered named UNIX socket transport module. [ 0.861923] RPC: Registered udp transport module. [ 0.863880] RPC: Registered tcp transport module. [ 0.865790] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.868414] NET: Registered protocol family 44 [ 0.872272] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.874808] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.877190] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.879655] PCI: CLS 0 bytes, default 64 [ 0.881452] Unpacking initramfs... [ 2.374473] debug: unmapping init [mem 0xffff9546bcc64000-0xffff9546bffcffff] [ 2.378378] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.380675] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.383585] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.891792] Initialise system trusted keyrings [ 2.893408] Key type blacklist registered [ 2.894950] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.903981] zbud: loaded [ 2.907743] *** VALIDATE nfs *** [ 2.908953] *** VALIDATE nfs4 *** [ 2.923945] pstore: using deflate compression [ 2.929791] Platform Keyring initialized [ 3.072048] NET: Registered protocol family 38 [ 3.073954] Key type asymmetric registered [ 3.075483] Asymmetric key parser 'x509' registered [ 3.077327] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.080366] io scheduler mq-deadline registered [ 3.082144] io scheduler kyber registered [ 3.083851] io scheduler bfq registered [ 3.085678] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.088763] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.090874] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.093990] ACPI: Power Button [PWRF] [ 3.099251] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.106129] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.115674] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.145352] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.175658] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.180546] Non-volatile memory driver v1.3 [ 3.182253] Linux agpgart interface v0.103 [ 3.213446] virtio_blk virtio1: [vda] 145808 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.218344] vda: detected capacity change from 0 to 74653696 [ 3.236314] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.241193] vdb: detected capacity change from 0 to 1073741824 [ 3.248303] libphy: Fixed MDIO Bus: probed [ 3.257836] usbcore: registered new interface driver usbserial_generic [ 3.260255] usbserial: USB Serial support registered for generic [ 3.261685] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.265726] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.267653] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.270439] mousedev: PS/2 mouse device common for all mice [ 3.272925] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.273242] rtc_cmos 00:05: RTC can wake from S4 [ 3.283370] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.283574] rtc_cmos 00:05: registered as rtc0 [ 3.287829] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.288040] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.289895] intel_pstate: CPU model not supported [ 3.295105] hid: raw HID events driver (C) Jiri Kosina [ 3.297325] usbcore: registered new interface driver usbhid [ 3.299606] usbhid: USB HID core driver [ 3.301305] drop_monitor: Initializing network drop monitor service [ 3.304013] Initializing XFRM netlink socket [ 3.305953] NET: Registered protocol family 10 [ 3.308698] Segment Routing with IPv6 [ 3.310029] NET: Registered protocol family 17 [ 3.312297] mpls_gso: MPLS GSO support [ 3.317359] RAS: Correctable Errors collector initialized. [ 3.318826] AVX version of gcm_enc/dec engaged. [ 3.320213] AES CTR mode by8 optimization enabled [ 3.375055] sched_clock: Marking stable (3375029203, 0)->(4408434851, -1033405648) [ 3.377594] registered taskstats version 1 [ 3.379349] Loading compiled-in X.509 certificates [ 3.380650] zswap: loaded using pool lzo/zbud [ 3.401188] Key type big_key registered [ 3.411080] Key type encrypted registered [ 3.413580] ima: No TPM chip found, activating TPM-bypass! [ 3.416205] ima: Allocated hash algorithm: sha1 [ 3.418354] ima: No architecture policies found [ 3.420608] evm: Initialising EVM extended attributes: [ 3.422879] evm: security.selinux [ 3.424196] evm: security.ima [ 3.425636] evm: security.capability [ 3.427373] evm: HMAC attrs: 0x1 [ 3.432203] rtc_cmos 00:05: setting system clock to 2026-07-20 21:47:23 UTC (1784584043) [ 3.439781] debug: unmapping init [mem 0xffffffff8ca03000-0xffffffff8cbfffff] [ 3.443559] debug: unmapping init [mem 0xffffffff8b782000-0xffffffff8ba58fff] [ 3.455567] Write protecting the kernel read-only data: 28672k [ 3.459569] debug: unmapping init [mem 0xffffffff89e03000-0xffffffff89ffffff] [ 3.462297] debug: unmapping init [mem 0xffffffff8a714000-0xffffffff8a7fffff] [ 3.504732] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.511340] systemd[1]: Detected virtualization kvm. [ 3.513243] systemd[1]: Detected architecture x86-64. [ 3.515126] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.540259] systemd[1]: No hostname configured. [ 3.541952] systemd[1]: Set hostname to . [ 3.543932] random: systemd: uninitialized urandom read (16 bytes read) [ 3.546333] systemd[1]: Initializing machine ID from random generator. [ 3.691658] random: systemd: uninitialized urandom read (16 bytes read) [ 3.694834] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.700186] random: systemd: uninitialized urandom read (16 bytes read) [ 3.702538] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.707912] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Timers. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.447886] device-mapper: uevent: version 1.0.3 [ 4.451166] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.189484] virtio_net virtio0 ens2: renamed from eth0 [ 5.288287] scsi host0: ata_piix [ 5.317249] scsi host1: ata_piix [ 5.318859] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.321559] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.996373] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.934371] random: crng init done [ 9.935228] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.408399] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.529182] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.767464] SELinux: Disabled at runtime. [ 11.832590] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.839867] systemd[1]: Detected virtualization kvm. [ 11.841665] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.315239] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.318665] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.323453] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.327531] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.330687] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.338512] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.349783] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ 12.495554] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.823261] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.158629] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.170409] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.345826] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.365639] EDAC sbridge: Ver: 1.1.2 [ 14.567418] Key type dns_resolver registered [ 14.881669] NFS: Registering the id_resolver key type [ 14.884045] Key type id_resolver registered [ 14.885965] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg151-client login: [ 63.048448] libcfs: loading out-of-tree module taints kernel. [ 63.227866] Key type ._llcrypt registered [ 63.241184] Key type .llcrypt registered [ 64.054591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 64.077281] alg: No test for adler32 (adler32-zlib) [ 65.654677] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 66.429193] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 68.167193] Key type lgssc registered [ 70.470562] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.117624] hrtimer: interrupt took 3760790 ns [ 199.278270] Lustre: Mounted lustre-client [ 204.555631] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 218.981101] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing check_logdir /tmp/testlogs/ [ 224.736840] Lustre: lustre-OST0000-osc-ffff95471a51c800: disconnect after 23s idle [ 225.291878] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing yml_node [ 229.431643] Lustre: DEBUG MARKER: Client: 2.17.54.183 [ 231.993432] Lustre: DEBUG MARKER: MDS: 2.17.54.183 [ 234.935714] Lustre: DEBUG MARKER: OSS: 2.17.54.183 [ 236.751918] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jul 20 17:51:15 EDT 2026 [ 254.759815] Lustre: Unmounted lustre-client [ 283.219552] Key type lgssc unregistered [ 283.568820] LNet: 8384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.579581] LNetError: 8384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 283.600140] LNet: Removed LNI 192.168.201.51@tcp [ 284.567514] Key type .llcrypt unregistered [ 284.570658] Key type ._llcrypt unregistered [ 312.748865] Key type ._llcrypt registered [ 312.750771] Key type .llcrypt registered [ 312.966115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 312.982535] alg: No test for adler32 (adler32-zlib) [ 314.020327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 314.104809] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 314.169628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 318.626359] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 318.740195] LNet: 9263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.750308] LNetError: 9263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 318.778440] LNet: Removed LNI 192.168.201.51@tcp [ 319.470751] Key type .llcrypt unregistered [ 319.477369] Key type ._llcrypt unregistered [ 327.274693] Key type ._llcrypt registered [ 327.278235] Key type .llcrypt registered [ 328.947768] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 17:52:47 (1784584367) [ 329.678466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 329.694545] alg: No test for adler32 (adler32-zlib) [ 330.660889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.773795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.823452] LNet: 9679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.873983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.967258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 330.998912] LNet: 9686:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 339.319287] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 17:52:57 (1784584377) [ 339.485433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.534044] LNet: 10262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.568284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.720351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 346.702664] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 17:53:05 (1784584385) [ 346.811519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.844643] LNet: 10849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.869284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.987509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 354.885952] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 17:53:13 (1784584393) [ 355.007231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.042277] LNet: 11429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.090671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.264960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 364.274213] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 17:53:22 (1784584402) [ 364.769366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.812458] LNet: 12022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.841790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.954383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 365.569670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 373.483143] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 17:53:31 (1784584411) [ 373.593336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 373.641681] LNet: 12605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 373.673181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.809526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 374.257356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 374.858393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 382.770921] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 17:53:41 (1784584421) [ 383.059457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.105671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.205288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 391.929107] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 17:53:50 (1784584430) [ 392.284863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.314831] LNet: 13795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.320250] LNet: 13795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 392.339735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.401232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 392.431101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 392.986391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 393.050566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 393.667442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 393.717579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.358799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 394.414496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 394.886746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 394.935800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 395.613172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 395.669077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.226106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 396.288964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.981374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 397.011965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 397.605315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 397.682569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 398.380886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 398.469197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 399.308574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 399.352193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 400.034362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 400.107393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 400.802597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 400.841205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 410.413634] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 17:54:08 (1784584448) [ 411.051631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.114692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.229355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 411.272419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 419.611951] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 17:54:18 (1784584458) [ 420.033190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.094583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.200063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 420.250594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 426.888782] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 17:54:25 (1784584465) [ 427.222856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.263159] LNet: 15652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.278822] LNet: 15652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 427.306686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.421420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 427.468551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 434.705623] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 17:54:33 (1784584473) [ 435.179808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.278952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.430938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 435.474646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 443.076284] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 17:54:41 (1784584481) [ 443.224225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.283120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.431402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 443.530732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 450.014794] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 17:54:48 (1784584488) [ 450.280977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.331180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.424577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 450.468748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 458.018972] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 17:54:56 (1784584496) [ 458.259156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.333196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.454792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.888964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.659321] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 17:55:04 (1784584504) [ 465.872035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.919507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.133545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.307512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 473.362391] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 17:55:12 (1784584512) [ 473.458566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.517668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.666879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.927475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 481.407605] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 17:55:20 (1784584520) [ 481.779601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.827854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.483054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.842881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.184329] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 17:55:28 (1784584528) [ 490.432409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.485597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.567777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.637662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.296533] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 17:55:36 (1784584536) [ 497.394749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.434207] LNet: 21070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.442732] LNet: 21070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 497.485560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.591278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 497.656845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 504.160797] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 17:55:42 (1784584542) [ 504.249208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.300690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.403473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 511.585946] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 17:55:50 (1784584550) [ 511.870669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.925310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.017670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 519.572318] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 17:55:58 (1784584558) [ 519.945267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.998861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.157571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 520.723690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 528.343131] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 17:56:07 (1784584567) [ 528.430647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.484881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.566777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 528.609952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 535.216851] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 17:56:14 (1784584574) [ 535.445710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.483815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.593391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 535.657653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 542.336244] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 17:56:20 (1784584580) [ 542.564743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.606849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.695914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 542.776316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 549.829395] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 17:56:28 (1784584588) [ 549.941295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.989323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.092431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 550.148435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 557.003388] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 17:56:35 (1784584595) [ 557.141617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.202416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.269351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 557.389375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 557.410492] LNetError: 25799:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 557.461116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 557.501562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 564.810794] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 17:56:43 (1784584603) [ 564.895748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.932611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.046569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GAeAfQ4NRh [ 565.145741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GAeAfQ4NRh [ 565.237928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GAeAfQ4NRh [ 572.354407] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 17:56:50 (1784584610) [ 572.749965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.788609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.853880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 572.896418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 572.939682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 579.420401] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 17:56:57 (1784584617) [ 579.593938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.667085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.812346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 579.844242] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 581.564834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 582.888026] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 584.711230] Key type lgssc registered [ 586.279274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 590.574336] LNetError: 27560:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.51@tcp99 has no available nets [ 593.821234] Key type lgssc unregistered [ 594.177404] LNetError: 28669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.198084] LNet: Removed LNI 192.168.201.51@tcp [ 595.068800] Key type .llcrypt unregistered [ 595.073876] Key type ._llcrypt unregistered [ 602.698636] Key type ._llcrypt registered [ 602.702414] Key type .llcrypt registered [ 604.409706] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 17:57:22 (1784584642) [ 604.850621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 604.866226] alg: No test for adler32 (adler32-zlib) [ 605.854829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.928915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 608.941437] LNet: Message forwarding will be enabled in 2s [ 611.999130] LNet: Message forwarding enabled [ 612.471459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 612.503325] LNet: Message forwarding will stop in 2s to 6s [ 617.133596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 617.838058] LNet: Message forwarding disabled [ 619.587486] LNet: Message forwarding will be enabled in 2s [ 621.939906] LNet: Message forwarding enabled [ 626.975586] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 17:57:45 (1784584665) [ 627.227371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.268618] LNet: 30074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.534693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.634793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 635.169662] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 17:57:53 (1784584673) [ 635.774286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.790879] LNet: 30659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.831679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.886488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 635.949118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 635.985337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 636.026394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 636.053429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 636.083423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 636.109901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 636.139603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 636.168629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 636.211464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 636.270472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 636.317531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 636.372609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 636.420595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 636.484613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 636.533847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 636.572898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 636.632356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 636.733256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 636.752653] LNetError: 30702:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 642.944961] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 17:58:01 (1784584681) [ 643.338353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.381863] LNet: 31273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.416981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.519364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 650.735954] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 17:58:09 (1784584689) [ 652.419466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg151-server [ 654.012852] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 17:58:12 (1784584692) [ 654.103566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.126145] LNet: 32198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.150427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.245909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 654.271405] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 655.313340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[52-56]@tcp [ 655.362153] LNet: 32337:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 655.415950] LNetError: 32206:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.53@tcp is being used as a gateway but routing feature is not turned on [ 655.445967] LNetError: 32206:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.53@tcp has gone from up to down [ 656.419487] LNetError: 32206:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.55@tcp is being used as a gateway but routing feature is not turned on [ 656.426058] LNetError: 32206:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 657.469965] LNetError: 32206:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.53@tcp is being used as a gateway but routing feature is not turned on [ 657.486065] LNetError: 32206:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 659.498852] LNetError: 32206:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.55@tcp is being used as a gateway but routing feature is not turned on [ 659.509823] LNetError: 32206:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 662.477373] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 17:58:21 (1784584701) [ 663.594956] LNetError: 32206:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.52@tcp is being used as a gateway but routing feature is not turned on [ 663.614033] LNetError: 32206:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 17 previous similar messages [ 664.180760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg151-server [ 665.826567] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 17:58:24 (1784584704) [ 666.135274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.179439] LNet: 33274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.187913] LNetError: 33274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.227382] LNet: Removed LNI 192.168.201.51@tcp [ 666.267735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.325498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 666.361490] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 667.241409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[52-57/2]@tcp [ 667.264304] LNet: 33391:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 667.271243] LNet: 33391:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 667.294813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[52-57/2]@tcp [ 667.299212] LNetError: 33278:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.56@tcp has gone from up to down [ 667.339058] LNetError: 33278:0:(lib-lnet.h:1272:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 674.721556] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 17:58:33 (1784584713) [ 674.870076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.897259] LNet: 33976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.912507] LNetError: 33976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.949455] LNet: Removed LNI 192.168.201.51@tcp [ 674.976626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.120892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 675.175875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.208676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.268246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 675.360292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.389730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.429845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 675.504811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.536843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.605305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 675.713821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.811311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.889475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 676.013444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.058151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.150878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 676.210543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.285691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.352876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 676.441978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.476935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.539152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 676.627117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.673481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.731625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 676.800644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.834380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.942402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.009267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.054252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.119585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 677.207521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.247894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.331544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 684.092480] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 17:58:42 (1784584722) [ 686.223110] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg151-server [ 687.815339] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 17:58:46 (1784584726) [ 690.005411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg151-server [ 691.813286] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 17:58:50 (1784584730) [ 691.917545] LNet: 35380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.922188] LNet: 35380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 692.494074] Key type .llcrypt unregistered [ 692.495245] Key type ._llcrypt unregistered [ 698.562229] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 698.729479] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 698.746470] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 698.779893] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 701.277956] Key type ._llcrypt registered [ 701.280710] Key type .llcrypt registered [ 701.421365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 701.436214] alg: No test for adler32 (adler32-zlib) [ 702.436421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.479146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.500495] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 703.172672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 703.189763] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 704.767404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 704.799447] LNet: Removed LNI 10.1.2.3@tcp [ 704.914329] LNet: 36110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.919646] LNetError: 36110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.932784] LNet: Removed LNI 192.168.201.51@tcp [ 705.596470] Key type .llcrypt unregistered [ 705.599943] Key type ._llcrypt unregistered [ 716.874400] Key type ._llcrypt registered [ 716.879497] Key type .llcrypt registered [ 718.204954] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 17:59:16 (1784584756) [ 718.813671] Key type .llcrypt unregistered [ 718.817674] Key type ._llcrypt unregistered [ 725.151038] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 725.352645] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 725.365408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 725.374450] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.931475] Key type ._llcrypt registered [ 726.933454] Key type .llcrypt registered [ 727.151442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.168595] alg: No test for adler32 (adler32-zlib) [ 728.143700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.200841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.255811] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 729.143710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 729.164497] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 732.326887] LNet: 37590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.335262] LNetError: 37590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 732.339836] LNet: Removed LNI 10.1.2.3@tcp [ 732.349873] LNet: Removed LNI 192.168.201.51@tcp [ 733.088570] Key type .llcrypt unregistered [ 733.098732] Key type ._llcrypt unregistered [ 744.291592] Key type ._llcrypt registered [ 744.293544] Key type .llcrypt registered [ 745.585807] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 17:59:44 (1784584784) [ 746.299113] Key type .llcrypt unregistered [ 746.302138] Key type ._llcrypt unregistered [ 753.232170] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 753.428644] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 753.446976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 753.460152] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 756.261409] Key type ._llcrypt registered [ 756.263610] Key type .llcrypt registered [ 756.459925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 756.475847] alg: No test for adler32 (adler32-zlib) [ 757.404579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.448985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 757.470240] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 759.115818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 759.134815] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.927987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 760.950623] LNet: Removed LNI 10.1.2.3@tcp [ 760.964356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 760.991795] LNet: Removed LNI 10.1.2.31@tcp [ 761.010233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 761.030816] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.607685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 761.628238] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 762.464218] LNet: 39512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.469724] LNetError: 39512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 762.476371] LNet: Removed LNI 10.1.2.31@tcp [ 762.486842] LNet: Removed LNI 10.1.2.3@tcp [ 763.075529] Key type .llcrypt unregistered [ 763.083648] Key type ._llcrypt unregistered [ 773.711159] Key type ._llcrypt registered [ 773.716673] Key type .llcrypt registered [ 774.876475] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 18:00:13 (1784584813) [ 775.296446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 775.309426] alg: No test for adler32 (adler32-zlib) [ 776.187354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.212432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.224994] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 776.909722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.944172] LNet: 40465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.952008] LNetError: 40465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.978515] LNet: Removed LNI 192.168.201.51@tcp [ 776.998047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.047857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.51@tcp [ 777.075863] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 777.745586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.51@tcp [ 777.781591] LNet: Removed LNI 192.168.201.51@tcp [ 777.834529] LNet: 40600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.400541] Key type .llcrypt unregistered [ 778.402569] Key type ._llcrypt unregistered [ 793.152336] Key type ._llcrypt registered [ 793.159701] Key type .llcrypt registered [ 794.829641] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 18:00:33 (1784584833) [ 797.274047] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg151-server [ 799.374583] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 18:00:37 (1784584837) [ 800.108257] Key type .llcrypt unregistered [ 800.111685] Key type ._llcrypt unregistered [ 807.126786] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 807.274670] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 807.287202] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 807.313881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 809.933281] Key type ._llcrypt registered [ 809.938516] Key type .llcrypt registered [ 810.152504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.178573] alg: No test for adler32 (adler32-zlib) [ 811.121890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.197904] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 811.982361] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 813.723543] LNet: 42438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.728863] LNetError: 42438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.734579] LNet: Removed LNI 192.168.201.51@tcp1 [ 813.753936] LNet: Removed LNI 10.1.2.3@tcp2 [ 814.152807] Key type .llcrypt unregistered [ 814.154638] Key type ._llcrypt unregistered [ 825.683378] Key type ._llcrypt registered [ 825.693148] Key type .llcrypt registered [ 827.145390] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 18:01:05 (1784584865) [ 827.709815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.722717] alg: No test for adler32 (adler32-zlib) [ 828.692994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.787392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 828.825569] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 829.632908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.643326] LNet: 43421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.649786] LNetError: 43421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.671699] LNet: Removed LNI 192.168.201.51@tcp [ 829.690694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.765417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 829.786563] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 836.750502] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 18:01:15 (1784584875) [ 837.044513] LNet: 44013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.054262] LNetError: 44013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.068181] LNet: Removed LNI 192.168.201.51@tcp [ 837.652495] Key type .llcrypt unregistered [ 837.656503] Key type ._llcrypt unregistered [ 846.296001] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 846.484924] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 846.494905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 846.509510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 849.175523] Key type ._llcrypt registered [ 849.178332] Key type .llcrypt registered [ 849.412201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 849.429424] alg: No test for adler32 (adler32-zlib) [ 850.423423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.522603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 850.562276] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 851.212296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 851.227039] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 852.973868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.985547] LNet: 44756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.990438] LNetError: 44756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 852.996252] LNet: Removed LNI 10.1.2.3@tcp [ 853.002790] LNet: Removed LNI 192.168.201.51@tcp [ 853.018758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.068772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 853.091632] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 853.094819] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.379504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.396569] LNet: Removed LNI 10.1.2.3@tcp [ 853.406873] LNet: Removed LNI 192.168.201.51@tcp [ 853.419651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.470334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 853.487360] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 853.491162] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.982191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 854.008473] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 854.684769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.700975] LNet: 44945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.705688] LNet: 44945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 854.712277] LNetError: 44945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 854.716990] LNetError: 44945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 854.725188] LNet: Removed LNI 10.1.2.3@tcp [ 854.726995] LNet: Removed LNI 192.168.201.51@tcp [ 854.741263] LNet: Removed LNI 192.168.201.51@tcp2 [ 854.763637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.823432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 854.887292] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 854.892919] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 854.898492] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 861.466370] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 18:01:40 (1784584900) [ 861.696272] LNet: 45542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.714514] LNetError: 45542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 861.720850] LNetError: 45542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 861.732256] LNet: Removed LNI 10.1.2.3@tcp [ 861.736203] LNet: Removed LNI 192.168.201.51@tcp [ 861.756229] LNet: Removed LNI 192.168.201.51@tcp2 [ 862.535413] Key type .llcrypt unregistered [ 862.541524] Key type ._llcrypt unregistered [ 870.083910] Key type ._llcrypt registered [ 870.085916] Key type .llcrypt registered [ 870.496296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.542437] alg: No test for adler32 (adler32-zlib) [ 871.547633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.642293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.696058] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 872.822703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.862865] LNet: 45938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.875690] LNetError: 45938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.910213] LNet: Removed LNI 192.168.201.51@tcp [ 872.951584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.040817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 873.077083] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 874.032889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.070282] LNet: 45969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.075510] LNetError: 45969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.120337] LNet: Removed LNI 192.168.201.51@tcp [ 874.176327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.300498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 874.351569] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 875.150907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 875.188350] LNet: 45999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.193958] LNetError: 45999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.204963] LNet: Removed LNI 192.168.201.51@tcp [ 875.240143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 875.338411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 875.366606] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 876.288436] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 876.456307] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 876.460788] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 876.465419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 878.927451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.961975] LNet: 46076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.968885] LNetError: 46076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.988418] LNet: Removed LNI 192.168.201.51@tcp [ 879.025470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.150721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 879.192226] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 888.434442] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 18:02:06 (1784584926) [ 888.562695] LNet: 46666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.570035] LNetError: 46666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.590513] LNet: Removed LNI 192.168.201.51@tcp [ 889.460942] Key type .llcrypt unregistered [ 889.464389] Key type ._llcrypt unregistered [ 897.199994] Key type ._llcrypt registered [ 897.206313] Key type .llcrypt registered [ 897.349800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.363543] alg: No test for adler32 (adler32-zlib) [ 898.367558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.432755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 898.457079] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 899.851233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.871848] LNet: 47072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.877335] LNetError: 47072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.898055] LNet: Removed LNI 192.168.201.51@tcp [ 899.925426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.068905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 900.131055] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 908.287470] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:02:26 (1784584946) [ 908.843637] LNet: 47667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.855638] LNetError: 47667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.884096] LNet: Removed LNI 192.168.201.51@tcp [ 909.584834] Key type .llcrypt unregistered [ 909.596174] Key type ._llcrypt unregistered [ 916.436611] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 916.647579] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 916.651913] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 916.655691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 919.156528] Key type ._llcrypt registered [ 919.159216] Key type .llcrypt registered [ 919.313518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 919.325297] alg: No test for adler32 (adler32-zlib) [ 920.279813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.384176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 920.438220] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 921.521483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.563988] LNet: 48116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.577469] LNetError: 48116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.612687] LNet: Removed LNI 192.168.201.51@tcp [ 921.673962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.724826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 921.744697] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 928.404524] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 18:02:47 (1784584967) [ 928.630025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.662767] LNet: 48714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.669423] LNetError: 48714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.682363] LNet: Removed LNI 192.168.201.51@tcp [ 928.708652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.780858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 935.883286] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 18:02:54 (1784584974) [ 936.344738] LNet: 49296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.952709] Key type .llcrypt unregistered [ 936.958665] Key type ._llcrypt unregistered [ 943.743887] Key type ._llcrypt registered [ 943.746319] Key type .llcrypt registered [ 943.920287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 943.935875] alg: No test for adler32 (adler32-zlib) [ 944.855302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.918770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 944.937608] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 945.733695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.758520] LNet: 49693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.772592] LNetError: 49693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.795763] LNet: Removed LNI 192.168.201.51@tcp [ 945.823461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.936465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 945.977450] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 952.730709] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 18:03:11 (1784584991) [ 952.880600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.921225] LNet: 50280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.925741] LNetError: 50280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.937899] LNet: Removed LNI 192.168.201.51@tcp [ 952.956190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.039786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 959.257581] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 18:03:17 (1784584997) [ 960.974424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 962.802380] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 18:03:21 (1784585001) [ 964.299474] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 966.046305] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 18:03:24 (1784585004) [ 967.755756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 969.413424] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 18:03:28 (1784585008) [ 970.853547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 972.793676] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 18:03:31 (1784585011) [ 974.258689] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 976.437771] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 18:03:34 (1784585014) [ 976.570969] LNet: 52333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.152986] Key type .llcrypt unregistered [ 977.155186] Key type ._llcrypt unregistered [ 983.833634] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 984.039176] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 984.043111] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 984.057842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 986.711647] Key type ._llcrypt registered [ 986.718665] Key type .llcrypt registered [ 986.981649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 987.001703] alg: No test for adler32 (adler32-zlib) [ 987.915689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.981750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 987.997663] LNet: Added LNI 192.168.201.51@tcp [8/512/0/180] [ 988.002024] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 988.003650] LNet: 52661:0:(api-ni.c:2929:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 988.068486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.078075] LNet: 52680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.082122] LNetError: 52680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 988.085190] LNet: Removed LNI 10.1.2.3@tcp [ 988.087949] LNetError: 52680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.100838] LNet: Removed LNI 192.168.201.51@tcp [ 988.120606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.209483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 988.291385] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 988.473211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.541522] LNet: 52704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.549817] LNetError: 52704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.582609] LNet: Removed LNI 192.168.201.51@tcp [ 988.623330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.747436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 988.800254] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 988.806046] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 988.813132] LNet: 52713:0:(api-ni.c:2929:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 988.966452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.988186] LNet: 52731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.006253] LNetError: 52731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 989.025087] LNet: Removed LNI 10.1.2.3@tcp [ 989.029874] LNetError: 52731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.065772] LNet: Removed LNI 192.168.201.51@tcp [ 989.091564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.217695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 989.297868] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 989.311926] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 989.313724] LNet: 52739:0:(api-ni.c:2929:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 989.692661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.722422] LNet: 52765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.728173] LNetError: 52765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 989.733834] LNet: Removed LNI 10.1.2.3@tcp [ 989.736483] LNetError: 52765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.750691] LNet: Removed LNI 192.168.201.51@tcp [ 989.813114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.920369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 989.962301] LNet: Added LNI 192.168.201.51@tcp [8/256/0/182] [ 989.968582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 990.117552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.151875] LNet: 52793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.165736] LNetError: 52793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 990.173809] LNet: Removed LNI 10.1.2.3@tcp [ 990.176740] LNetError: 52793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.200607] LNet: Removed LNI 192.168.201.51@tcp [ 990.216602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.312586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 990.349763] LNet: Added LNI 192.168.201.51@tcp [8/256/0/182] [ 990.355405] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 990.357753] LNet: 52801:0:(api-ni.c:2929:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 990.472808] LNet: Removed LNI 10.1.2.3@tcp [ 990.487100] LNet: Removed LNI 192.168.201.51@tcp [ 991.800778] Key type .llcrypt unregistered [ 991.810844] Key type ._llcrypt unregistered [ 1005.637400] Key type ._llcrypt registered [ 1005.640480] Key type .llcrypt registered [ 1007.205140] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 18:04:05 (1784585045) [ 1007.891371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1007.922023] alg: No test for adler32 (adler32-zlib) [ 1008.873670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.913577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1008.927779] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1009.495126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.51 [ 1009.508889] LNet: 53776:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1009.515659] LNet: Removed LNI 192.168.201.51@tcp [ 1015.882300] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 18:04:14 (1784585054) [ 1016.044051] LNet: 54341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.060992] LNetError: 54341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.107361] LNet: Removed LNI 192.168.201.51@tcp [ 1016.859488] Key type .llcrypt unregistered [ 1016.862335] Key type ._llcrypt unregistered [ 1023.949458] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1024.188444] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1027.121739] Key type ._llcrypt registered [ 1027.125316] Key type .llcrypt registered [ 1027.356916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.373184] alg: No test for adler32 (adler32-zlib) [ 1028.879269] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1035.386818] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:04:33 (1784585073) [ 1035.578500] LNet: 55259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.583956] LNetError: 55259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1035.627625] LNet: Removed LNI 10.1.2.3@tcp [ 1036.286744] Key type .llcrypt unregistered [ 1036.288468] Key type ._llcrypt unregistered [ 1042.880168] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1043.226533] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1043.233617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1043.240532] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1045.009917] Key type ._llcrypt registered [ 1045.018995] Key type .llcrypt registered [ 1045.216939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1045.241744] alg: No test for adler32 (adler32-zlib) [ 1046.261953] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1053.431979] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:04:52 (1784585092) [ 1055.001550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1057.148842] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:04:55 (1784585095) [ 1057.341679] LNet: 56433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.353479] LNetError: 56433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1057.397573] LNet: Removed LNI 10.1.2.3@tcp [ 1058.087656] Key type .llcrypt unregistered [ 1058.100770] Key type ._llcrypt unregistered [ 1064.650533] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1064.911225] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1064.921863] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1064.939064] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1067.742049] Key type ._llcrypt registered [ 1067.747682] Key type .llcrypt registered [ 1067.940167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1067.957522] alg: No test for adler32 (adler32-zlib) [ 1069.009946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.147745] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1078.756842] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:05:17 (1784585117) [ 1079.306243] LNet: 57669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.317355] LNetError: 57669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1079.353319] LNet: Removed LNI 10.1.2.3@tcp [ 1080.055577] Key type .llcrypt unregistered [ 1080.057475] Key type ._llcrypt unregistered [ 1086.774765] Key type ._llcrypt registered [ 1086.780484] Key type .llcrypt registered [ 1086.990919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1087.013775] alg: No test for adler32 (adler32-zlib) [ 1087.958665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1087.990808] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1096.019669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1096.757491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_interrupt -r 1 [ 1096.828947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_interrupt -r 1 [ 1096.914230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1096.968069] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1097.249628] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1097.295163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.322315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1097.355532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.415512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.444061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.501915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.547066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.595627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.626982] LNet: 58106:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1097.664380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.728568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.773819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.819315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.861929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.907124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.952525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1098.622498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_dropped -r 1 [ 1098.661823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_dropped -r 1 [ 1098.701632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1098.722592] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1098.728347] LNet: Skipped 1 previous similar message [ 1098.985277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.030968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1099.081907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1099.112732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1099.137835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1099.177810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1099.203790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1099.228323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1099.239414] LNet: 58216:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1099.259946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1099.296458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1099.329946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1099.364824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1099.399312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1099.456406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1099.506346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1100.141710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_aborted -r 1 [ 1100.216095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_aborted -r 1 [ 1100.285743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1100.316504] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1100.527683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1100.580147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1100.603354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1100.628657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1100.669648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.728920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.801498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.864638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.896745] LNet: 58325:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.922849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.979378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1101.032270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.069609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1101.132055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1101.184816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1101.234741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1101.910898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_no_route -r 1 [ 1101.937936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_no_route -r 1 [ 1101.965226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1102.182178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1102.206981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1102.250423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1102.282805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.306189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.332137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.350559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.374373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.425426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.483292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.543582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.580396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.630173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.681118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.732258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.219764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_timeout -r 1 [ 1103.250739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_timeout -r 1 [ 1103.297342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1103.322763] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1103.332278] LNet: Skipped 1 previous similar message [ 1103.597263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.623751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.649963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.681217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.728524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.746443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.771397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.810638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.833024] LNet: 58543:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.845448] LNet: 58543:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1103.863604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.911411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1103.944812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.973806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.009407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.051422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.089452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.718810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_error -r 1 [ 1104.762217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_error -r 1 [ 1104.804623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1104.998756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.027913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.068691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.108996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.148801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.189034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.217631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.249394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.276982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.313440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.350505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.407711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.436922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.477222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.507429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.822935] LNet: 58694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.850521] LNetError: 58694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.877743] LNet: Removed LNI 192.168.201.51@tcp [ 1106.355758] Key type .llcrypt unregistered [ 1106.358899] Key type ._llcrypt unregistered [ 1128.253886] Key type ._llcrypt registered [ 1128.255488] Key type .llcrypt registered [ 1129.656751] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:06:08 (1784585168) [ 1130.642674] Key type .llcrypt unregistered [ 1130.648364] Key type ._llcrypt unregistered [ 1138.178559] Key type ._llcrypt registered [ 1138.188316] Key type .llcrypt registered [ 1138.476069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1138.508791] alg: No test for adler32 (adler32-zlib) [ 1139.444755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1139.472741] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1147.749720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1159.551535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1159.667569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1159.716721] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1161.487549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_interrupt -r 1 [ 1161.518608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 1 [ 1161.557239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_interrupt -r 1 [ 1161.630100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 1 [ 1161.714568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_interrupt -r 1 [ 1161.787747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 1 [ 1161.832441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_interrupt -r 1 [ 1161.880942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 1 [ 1161.924823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1161.946955] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1161.954923] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1162.215990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.253401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1162.285827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.313971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.359358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.430649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.503947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.561545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.614649] LNet: 60161:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.666859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.725315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.848361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.926463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.979933] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1162.995577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.996609] LNet: Skipped 3 previous similar messages [ 1163.058276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.093866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1163.891546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_dropped -r 1 [ 1163.960968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 1 [ 1164.072934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_dropped -r 1 [ 1164.140903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 1 [ 1164.241416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_dropped -r 1 [ 1164.297419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 1 [ 1164.344460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_dropped -r 1 [ 1164.428562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 1 [ 1164.477186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1164.504348] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1164.925133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.985166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.044831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.114631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.183495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.241410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.327423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.383989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.430381] LNet: 60283:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.461840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.506609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.614479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.730485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.823362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.979246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.085606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.053970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_aborted -r 1 [ 1167.105219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 1 [ 1167.165245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_aborted -r 1 [ 1167.216911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 1 [ 1167.284565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_aborted -r 1 [ 1167.366869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 1 [ 1167.437502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_aborted -r 1 [ 1167.527036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 1 [ 1167.620985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1167.679174] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1167.702688] LNet: Skipped 2 previous similar messages [ 1168.064189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.114644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.188129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.276984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.398444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.483436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.568827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.722594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.761176] LNet: 60405:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.846979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.935592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.023024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.093786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.141559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.190589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.223082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1170.037217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_no_route -r 1 [ 1170.091567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 1 [ 1170.190445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_no_route -r 1 [ 1170.294945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 1 [ 1170.359589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_no_route -r 1 [ 1170.432421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 1 [ 1170.499424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_no_route -r 1 [ 1170.580440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 1 [ 1170.647395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1171.080826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.150641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.220929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.256907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.289780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.312566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.343314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.392997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.425096] LNet: 60527:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.443807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.497347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.533577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.561862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.591498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.619592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.645663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1172.243135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_timeout -r 1 [ 1172.312756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 1 [ 1172.390307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_timeout -r 1 [ 1172.464110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 1 [ 1172.538277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_timeout -r 1 [ 1172.601162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 1 [ 1172.644860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_timeout -r 1 [ 1172.709872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 1 [ 1172.784202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1172.841343] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1172.849624] LNet: Skipped 5 previous similar messages [ 1173.289738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.349178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.476133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.561932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.634606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.704079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.765773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.827489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.883334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.953078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.011103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.112833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.222693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.292197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.366203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.411993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_error -r 1 [ 1175.473520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_error -r 1 [ 1175.571301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_error -r 1 [ 1175.670146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_error -r 1 [ 1175.769869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_error -r 1 [ 1175.878228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_error -r 1 [ 1175.966980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_error -r 1 [ 1176.056784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_error -r 1 [ 1176.149757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1176.579619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.653663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.718644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.758706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.807788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.850830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.880162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.948948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.990804] LNet: 60769:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1177.009657] LNet: 60769:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1177.064280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.116271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.171086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.203277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.250656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.306895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.356919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1185.220926] LNet: 60846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1185.241791] LNetError: 60846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.271155] LNet: Removed LNI 192.168.201.51@tcp [ 1186.297150] LNetError: 60846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1186.323741] LNet: Removed LNI 192.168.201.51@tcp1 [ 1186.861252] Key type .llcrypt unregistered [ 1186.871707] Key type ._llcrypt unregistered [ 1205.352306] Key type ._llcrypt registered [ 1205.354855] Key type .llcrypt registered [ 1206.469582] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:07:25 (1784585245) [ 1207.267340] Key type .llcrypt unregistered [ 1207.270495] Key type ._llcrypt unregistered [ 1213.571606] Key type ._llcrypt registered [ 1213.574840] Key type .llcrypt registered [ 1213.789681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1213.821167] alg: No test for adler32 (adler32-zlib) [ 1214.854640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1214.886772] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1224.194418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1225.449889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_dropped -r 1 [ 1225.540492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_dropped -r 1 [ 1225.615407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1225.636850] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1226.091863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.138430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.210086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.283525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.329814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.387433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.441588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.475579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.506666] LNet: 62096:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.535912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.608374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.644565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.674074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.709375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.741843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.776598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1227.534352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_error -r 1 [ 1227.564507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_error -r 1 [ 1227.594171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1227.614944] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1227.839334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.858923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1227.879844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.899926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.941815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.986444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.030369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.085301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.110809] LNet: 62205:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.128137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.165267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.200164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.220933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.244154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.280942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.307451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1229.020549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_timeout -r 1 [ 1229.095164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_timeout -r 1 [ 1229.166149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1229.233887] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1229.567810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.613588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.657322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.687449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.747124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.778688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.810570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.864344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.884176] LNet: 62314:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.918941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.984927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.056616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.104205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.164882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.248659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.310740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1230.780224] LNet: 62355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.800780] LNetError: 62355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.838901] LNet: Removed LNI 192.168.201.51@tcp [ 1231.496402] Key type .llcrypt unregistered [ 1231.499161] Key type ._llcrypt unregistered [ 1252.444346] Key type ._llcrypt registered [ 1252.445881] Key type .llcrypt registered [ 1253.896579] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:08:12 (1784585292) [ 1254.748455] Key type .llcrypt unregistered [ 1254.751332] Key type ._llcrypt unregistered [ 1260.675761] Key type ._llcrypt registered [ 1260.681812] Key type .llcrypt registered [ 1261.025500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1261.071272] alg: No test for adler32 (adler32-zlib) [ 1262.100480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1262.129143] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1271.061916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1282.239751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1282.385178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1282.463949] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1284.624333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_dropped -r 1 [ 1284.660724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 1 [ 1284.693752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_dropped -r 1 [ 1284.763482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 1 [ 1284.843301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_dropped -r 1 [ 1284.917765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 1 [ 1284.992429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_dropped -r 1 [ 1285.056120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 1 [ 1285.158993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1285.203347] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1285.213915] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 1285.644392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1285.694551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1285.753323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1285.848186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1285.908727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1285.973981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1286.061117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1286.147808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1286.209371] LNet: 63822:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1286.239682] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1286.261861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1286.262679] LNet: Skipped 3 previous similar messages [ 1286.330704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1286.390447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1286.447822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1286.512790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1286.565682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1286.611698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1286.664315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1287.389900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_error -r 1 [ 1287.424977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_error -r 1 [ 1287.457919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_error -r 1 [ 1287.481820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_error -r 1 [ 1287.513357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_error -r 1 [ 1287.542529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_error -r 1 [ 1287.586410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_error -r 1 [ 1287.625660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_error -r 1 [ 1287.665189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1287.684950] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1287.984651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.015173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.069131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.126791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.186805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.233206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.307179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.341353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.380607] LNet: 63945:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.432932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.518596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.584260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.663553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.718593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.766762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.805627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1288.869819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1289.461438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_timeout -r 1 [ 1289.481177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_timeout -r 1 [ 1289.534235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_timeout -r 1 [ 1289.606641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_timeout -r 1 [ 1289.684547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_timeout -r 1 [ 1289.745994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_timeout -r 1 [ 1289.805599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_timeout -r 1 [ 1289.907906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_timeout -r 1 [ 1289.987730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.151@tcp [ 1290.025631] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1290.358782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1290.402810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1290.446488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1290.507518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1290.545664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1290.584952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1290.658924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1290.684546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1290.701708] LNet: 64069:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1290.717613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.743398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.784842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.835379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.890569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.927678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1290.965798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1291.020316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.655856] LNet: 64147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.667270] LNetError: 64147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1296.676367] LNet: Removed LNI 192.168.201.51@tcp [ 1296.695657] LNet: Removed LNI 192.168.201.51@tcp1 [ 1297.398179] Key type .llcrypt unregistered [ 1297.402308] Key type ._llcrypt unregistered [ 1317.436250] Key type ._llcrypt registered [ 1317.437868] Key type .llcrypt registered [ 1318.689673] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:09:17 (1784585357) [ 1320.365126] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1322.296309] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:09:20 (1784585360) [ 1323.606334] Key type .llcrypt unregistered [ 1323.608072] Key type ._llcrypt unregistered [ 1329.753764] Key type ._llcrypt registered [ 1329.755888] Key type .llcrypt registered [ 1329.979682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1329.989646] alg: No test for adler32 (adler32-zlib) [ 1331.048717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1331.118084] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1338.148591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1338.864294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e network_timeout -r 1 [ 1338.888588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e network_timeout -r 1 [ 1338.919795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1338.934077] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1339.167946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1339.198698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1339.231629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1339.257780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1339.283556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1339.303941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1339.330675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1339.359548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1339.360638] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1339.384636] LNet: 65690:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1339.405506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1339.433718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.471532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.499282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.531879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.575080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.823764] LNet: 65729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.835404] LNetError: 65729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1339.855375] LNet: Removed LNI 192.168.201.51@tcp [ 1340.330753] Key type .llcrypt unregistered [ 1340.335654] Key type ._llcrypt unregistered [ 1358.319981] Key type ._llcrypt registered [ 1358.327952] Key type .llcrypt registered [ 1358.665902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1358.711700] alg: No test for adler32 (adler32-zlib) [ 1359.701159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1359.764292] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1369.282236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1379.473761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.577515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1379.634483] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1381.123728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e network_timeout -r 1 [ 1381.182815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e network_timeout -r 1 [ 1381.267146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e network_timeout -r 1 [ 1381.348462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e network_timeout -r 1 [ 1381.440792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e network_timeout -r 1 [ 1381.516579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e network_timeout -r 1 [ 1381.592292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e network_timeout -r 1 [ 1381.667587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e network_timeout -r 1 [ 1381.750258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1381.833570] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1382.197685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1382.230574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1382.292515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1382.326674] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1382.342736] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 1382.366261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1382.421649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1382.471685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1382.536838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1382.571702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1382.606753] LNet: 66610:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1382.651438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1382.730779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1382.805993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.870678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1382.955866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1383.017928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1388.079082] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1388.091848] LNet: Skipped 2 previous similar messages [ 1388.623554] LNet: 66683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.636090] LNetError: 66683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.664050] LNet: Removed LNI 192.168.201.51@tcp [ 1389.671786] LNetError: 66683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1389.696623] LNet: Removed LNI 192.168.201.51@tcp1 [ 1390.517759] Key type .llcrypt unregistered [ 1390.521765] Key type ._llcrypt unregistered [ 1409.621305] Key type ._llcrypt registered [ 1409.623791] Key type .llcrypt registered [ 1410.952782] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:10:49 (1784585449) [ 1411.324534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1411.353548] alg: No test for adler32 (adler32-zlib) [ 1412.248662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.321542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1412.364154] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1413.169716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1413.184821] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1414.180659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1414.260747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1414.303519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1414.338247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1414.381769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1414.420568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1414.461520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.51@tcp [ 1414.480821] LNetError: 67784:0:(api-ni.c:4416:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1415.391454] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1415.391487] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1417.439505] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1418.295557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1421.535524] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1421.954603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1421.971820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1422.027875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.053054] LNet: 67891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.058859] LNetError: 67891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.064447] LNet: Removed LNI 192.168.201.51@tcp [ 1422.079947] LNet: Removed LNI 192.168.201.51@tcp1 [ 1422.110228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.178654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.203228] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1422.963919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1422.988435] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1423.907992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1423.956649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1424.044806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1424.100902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1424.136462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1424.171641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1424.239714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1424.282926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.51@tcp [ 1424.315914] LNetError: 68164:0:(api-ni.c:4416:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1425.247877] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1427.915539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1431.395026] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1431.417061] LNet: Skipped 1 previous similar message [ 1435.987423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1436.030813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1436.068271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1444.216970] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:11:22 (1784585482) [ 1444.771558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1444.797387] LNet: 68882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1444.803685] LNetError: 68882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1444.815681] LNetError: 68882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1444.833212] LNet: Removed LNI 192.168.201.51@tcp [ 1444.853258] LNet: Removed LNI 192.168.201.51@tcp1 [ 1444.878842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.948672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1444.973854] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1445.660697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1445.682608] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1446.521416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1446.566432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1446.589414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1446.613960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1446.645770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1446.688148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1446.713827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.51@tcp --health 0 [ 1446.848875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1446.944223] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.51@tcp [ 1447.969607] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1447.974719] LNet: Skipped 1 previous similar message [ 1462.512195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1462.813365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1462.847317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1462.929446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.51@tcp --health 500 [ 1463.106335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1468.915904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1469.023673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.057273] LNet: 69366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.073274] LNetError: 69366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.076456] LNetError: 69366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1469.094450] LNet: Removed LNI 192.168.201.51@tcp [ 1469.116793] LNet: Removed LNI 192.168.201.51@tcp1 [ 1469.150723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.268235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.340790] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1470.294990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1470.314947] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1471.465874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1471.503355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1471.591413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1471.623266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1471.662393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1471.721713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1471.761693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1471.794517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.51@tcp --health 0 [ 1472.287573] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1472.294019] LNet: Skipped 3 previous similar messages [ 1473.265214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1482.858679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1482.888856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1482.913842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1491.053449] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:12:09 (1784585529) [ 1491.713526] LNet: 70400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.724593] LNetError: 70400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.739542] LNetError: 70400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1491.751239] LNet: Removed LNI 192.168.201.51@tcp [ 1491.770654] LNet: Removed LNI 192.168.201.51@tcp1 [ 1492.675397] Key type .llcrypt unregistered [ 1492.679395] Key type ._llcrypt unregistered [ 1499.830399] Key type ._llcrypt registered [ 1499.832332] Key type .llcrypt registered [ 1500.141304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1500.157887] alg: No test for adler32 (adler32-zlib) [ 1501.375959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1501.458484] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1511.968453] Lustre: DEBUG MARKER: Initial discovery [ 1512.756994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.151@tcp [ 1515.918592] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1516.047651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1516.194584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.151@tcp [ 1516.258778] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1516.322971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1516.408230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.151@tcp -r 1 -m GET -l 3 [ 1516.511711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.151@tcp -r 1 -m GET -e local_error [ 1516.595840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.151@tcp -r 1 -m PUT -l 6 [ 1516.834386] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1519.243660] Lustre: DEBUG MARKER: Force oleg151-client.virtnet to discover 192.168.201.151@tcp (in background) [ 1519.324937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.151@tcp [ 1521.526526] Lustre: DEBUG MARKER: Force oleg151-server to discover 192.168.201.51@tcp [ 1522.401388] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1525.160547] Lustre: DEBUG MARKER: Wait for 71159 [ 1525.987234] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1527.509320] Lustre: DEBUG MARKER: Finished wait on 71159 [ 1527.539519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1527.580239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1527.806313] LNet: 71618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.810584] LNetError: 71618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.821635] LNet: Removed LNI 192.168.201.51@tcp [ 1528.231949] Key type .llcrypt unregistered [ 1528.237496] Key type ._llcrypt unregistered [ 1548.407311] Key type ._llcrypt registered [ 1548.409388] Key type .llcrypt registered [ 1549.756834] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:13:08 (1784585588) [ 1550.339388] Key type .llcrypt unregistered [ 1550.341469] Key type ._llcrypt unregistered [ 1557.600056] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1557.827018] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1557.832649] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1557.837393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1560.440913] Key type ._llcrypt registered [ 1560.444247] Key type .llcrypt registered [ 1560.651665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1560.669619] alg: No test for adler32 (adler32-zlib) [ 1561.632952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.694773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.733858] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1562.431319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1562.449555] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1571.121785] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:13:29 (1784585609) [ 1571.392222] LNet: 73697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.398887] LNetError: 73697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1571.406397] LNet: Removed LNI 10.1.2.3@tcp [ 1571.425546] LNet: Removed LNI 192.168.201.51@tcp [ 1571.985473] Key type .llcrypt unregistered [ 1571.988365] Key type ._llcrypt unregistered [ 1579.435268] Key type ._llcrypt registered [ 1579.438910] Key type .llcrypt registered [ 1579.667647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1579.682251] alg: No test for adler32 (adler32-zlib) [ 1580.675468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.815305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1580.862625] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1581.840269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1581.878633] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1583.484355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.51@tcp [ 1584.552891] LNetError: Unexpected error -22 connecting to 192.168.201.51@tcp at host 192.168.201.51:988 [ 1591.571743] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:13:50 (1784585630) [ 1591.790035] LNet: 74829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.815650] LNetError: 74829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1591.827206] LNet: Removed LNI 10.1.2.3@tcp [ 1591.840848] LNet: Removed LNI 192.168.201.51@tcp [ 1592.839443] Key type .llcrypt unregistered [ 1592.840692] Key type ._llcrypt unregistered [ 1599.931816] Key type ._llcrypt registered [ 1599.938912] Key type .llcrypt registered [ 1600.159650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1600.173420] alg: No test for adler32 (adler32-zlib) [ 1600.999486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.031369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1601.043532] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1601.598698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1601.618540] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 1602.451486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.51@tcp1 --nid 192.168.201.51@tcp2 [ 1606.557443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp1 192.168.201.51@tcp2 [ 1606.584801] LNetError: 75666:0:(lib-move.c:2066:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.51@tcp1 to 192.168.201.51@tcp2 - Route table may be misconfigured [ 1606.623679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp2 192.168.201.51@tcp1 [ 1612.985960] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:14:11 (1784585651) [ 1613.141507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.179604] LNet: 76237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1613.198264] LNetError: 76237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.218118] LNet: Removed LNI 192.168.201.51@tcp1 [ 1614.246459] LNet: Removed LNI 192.168.201.51@tcp2 [ 1614.290614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.354675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.386981] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1615.257652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1615.281987] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1616.115703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 1616.148953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.51@tcp [ 1616.186821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp1 192.168.201.51@tcp1 [ 1616.223126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.51@tcp -d 192.168.201.51@tcp -e network_timeout [ 1616.258694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -e network_timeout [ 1616.293902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -e network_timeout [ 1616.330034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -e network_timeout [ 1616.366836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1616.410908] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1616.424558] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1616.445861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1616.601797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1616.726958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1623.626373] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:14:21 (1784585661) [ 1623.757775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.790574] LNet: 77096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.798626] LNetError: 77096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.804484] LNetError: 77096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1623.808072] LNet: Removed LNI 192.168.201.51@tcp [ 1624.880736] LNet: Removed LNI 192.168.201.51@tcp1 [ 1624.912385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.011041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1625.307562] LNet: 77139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1625.718398] Key type .llcrypt unregistered [ 1625.719930] Key type ._llcrypt unregistered [ 1639.482706] Key type ._llcrypt registered [ 1639.490221] Key type .llcrypt registered [ 1641.525483] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:14:39 (1784585679) [ 1642.016358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1642.041659] alg: No test for adler32 (adler32-zlib) [ 1643.054718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.936417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1646.790580] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:14:45 (1784585685) [ 1647.071661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1647.131736] LNet: 78237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1647.188454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.273051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1647.297588] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1648.019246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1648.042105] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1649.408584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1649.472175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp1 [ 1649.590802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp1 [ 1658.459658] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:14:56 (1784585696) [ 1661.073461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg151-server [ 1663.623830] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:15:01 (1784585701) [ 1666.080448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg151-server [ 1668.219749] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:15:06 (1784585706) [ 1671.364218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg151-server [ 1673.733582] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:15:11 (1784585711) [ 1676.015499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg151-server [ 1677.862239] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:15:16 (1784585716) [ 1680.082450] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg151-server [ 1682.331258] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:15:20 (1784585720) [ 1684.218241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg151-server [ 1686.468712] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:15:24 (1784585724) [ 1688.578414] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg151-server [ 1690.326559] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:15:29 (1784585729) [ 1692.634077] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg151-server [ 1694.755375] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:15:33 (1784585733) [ 1696.967642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg151-server [ 1698.788987] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:15:37 (1784585737) [ 1698.975877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.019842] LNet: 82116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.032101] LNetError: 82116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.046062] LNet: Removed LNI 192.168.201.51@tcp [ 1699.071287] LNet: Removed LNI 192.168.201.51@tcp1 [ 1699.117218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.188521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1699.214488] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1700.011408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1700.123575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1700.385784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.419463] LNet: 82265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.438340] LNetError: 82265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.441442] LNetError: 82265:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1700.471347] LNet: Removed LNI 192.168.201.51@tcp [ 1700.510153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.614716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1700.652146] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1701.805166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1701.982785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.998068] LNetError: 82410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.009576] LNet: Removed LNI 192.168.201.51@tcp [ 1702.024571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.076742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.104850] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1702.835400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1702.942615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1703.179875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.210648] LNet: 82557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.218691] LNet: 82557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1704.293832] LNet: Removed LNI 192.168.201.51@tcp [ 1704.340884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.403545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1704.435658] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1705.254245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1705.408223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.430663] LNetError: 82703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.441729] LNetError: 82703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1705.456765] LNet: Removed LNI 192.168.201.51@tcp [ 1705.478253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.522312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1705.546750] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1706.467430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1706.602537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1706.784584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.818221] LNet: Removed LNI 192.168.201.51@tcp [ 1707.842226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.930299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1707.967320] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1708.690488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1708.877391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.894893] LNet: 82995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.898195] LNet: 82995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1709.928663] LNet: Removed LNI 192.168.201.51@tcp [ 1709.958727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.079890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.124889] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1710.964693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1711.033600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1711.219922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.257480] LNetError: 83143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1711.267708] LNetError: 83143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1712.299578] LNet: Removed LNI 192.168.201.51@tcp [ 1712.370067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.485959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1712.530652] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1713.571191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1713.809260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.923382] LNet: Removed LNI 192.168.201.51@tcp [ 1714.987467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.092090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1715.142315] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1715.896519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1716.013266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1716.416721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.490487] LNet: Removed LNI 192.168.201.51@tcp [ 1717.562961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.676869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1717.726358] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1718.729943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1720.378726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.439953] LNet: 83591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.458490] LNet: 83591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1720.472308] LNetError: 83591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.479207] LNetError: 83591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1720.502223] LNet: Removed LNI 192.168.201.51@tcp [ 1720.522369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.624230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1720.643258] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1721.508679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1721.571888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1721.853572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.920881] LNet: Removed LNI 192.168.201.51@tcp [ 1722.971069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.069565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1723.106610] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1724.370778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 1724.793751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.862895] LNet: Removed LNI 192.168.201.51@tcp [ 1725.920643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.039089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1726.091945] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1727.139233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1727.187158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.214212] LNet: Removed LNI 192.168.201.51@tcp [ 1727.233545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.286778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.325541] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1728.047636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1728.130532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.148917] LNet: Removed LNI 192.168.201.51@tcp [ 1728.161338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.231486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1728.332580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.365778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.416350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1728.539581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.577813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.624197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1728.718343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.755377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.802760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1728.841479] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1730.265286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.311306] LNet: Removed LNI 192.168.201.51@tcp [ 1730.333604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.446935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1738.921764] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:16:17 (1784585777) [ 1739.014732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.050494] LNet: 84896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.059498] LNet: 84896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1739.091494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.195823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1739.227633] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1740.941460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.978668] LNetError: 85044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.989537] LNetError: 85044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1741.010303] LNet: Removed LNI 192.168.201.51@tcp231 [ 1741.054436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.149344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1741.177847] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1741.183459] LNet: 85052:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1741.821750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1741.875893] LNet: Removed LNI 192.168.201.51@tcp231 [ 1741.943869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1741.990379] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1743.438458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.487686] LNet: Removed LNI 192.168.201.51@tcp231 [ 1743.535325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.705384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1743.759816] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/0] [ 1743.767635] LNet: 85204:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1744.255367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1744.311673] LNet: Removed LNI 192.168.201.51@tcp231 [ 1744.327487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1744.348812] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/0] [ 1746.261417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.329262] LNet: Removed LNI 192.168.201.51@tcp231 [ 1746.380816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.491678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1746.515549] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/60] [ 1746.520813] LNet: 85356:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1747.124387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1747.155256] LNet: Removed LNI 192.168.201.51@tcp231 [ 1747.205667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1747.246363] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/60] [ 1749.008489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.046614] LNet: Removed LNI 192.168.201.51@tcp231 [ 1749.073375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.175516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1749.218616] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1749.224042] LNet: 85507:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1749.834508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1749.874453] LNet: Removed LNI 192.168.201.51@tcp231 [ 1749.888954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1749.915398] LNet: Added LNI 192.168.201.51@tcp231 [8/256/0/180] [ 1760.344581] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:16:38 (1784585798) [ 1760.456104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.539196] LNet: Removed LNI 192.168.201.51@tcp231 [ 1760.567417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.689848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1760.751980] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1761.708732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1761.777711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1761.814607] LNet: Removed LNI 192.168.201.51@tcp232 [ 1761.831194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1761.853482] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1762.860366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1762.914681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1762.953918] LNet: Removed LNI 192.168.201.51@tcp232 [ 1762.994412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1763.036662] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1764.027760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1764.072780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1764.117630] LNet: Removed LNI 192.168.201.51@tcp232 [ 1764.143142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1764.188965] LNet: Added LNI 192.168.201.51@tcp232 [8/256/0/180] [ 1765.215763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1765.255659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1765.285900] LNet: Removed LNI 192.168.201.51@tcp232 [ 1765.303665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1772.335298] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:16:51 (1784585811) [ 1772.824987] LNet: 87347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.827958] LNet: 87347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1773.344503] Key type .llcrypt unregistered [ 1773.351351] Key type ._llcrypt unregistered [ 1780.370785] Key type ._llcrypt registered [ 1780.375787] Key type .llcrypt registered [ 1780.604482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1780.632903] alg: No test for adler32 (adler32-zlib) [ 1781.612864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1781.651491] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1789.434572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1798.874794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.922352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1798.953379] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1800.735065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_interrupt -r 2 [ 1800.780512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 2 [ 1800.818742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_interrupt -r 2 [ 1800.849275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 2 [ 1800.882314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_interrupt -r 2 [ 1800.947132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_interrupt -r 2 [ 1801.014405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_interrupt -r 2 [ 1801.061171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_interrupt -r 2 [ 1801.104904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1801.126508] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1801.132602] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1801.515923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1801.557201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1801.592725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1801.651731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1801.694030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1801.747343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1801.795522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1801.826317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1801.840499] LNet: 88010:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1801.855466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1801.893834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1801.923655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1801.960036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1802.001934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1802.031557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1802.683412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_dropped -r 2 [ 1802.727958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 2 [ 1802.763741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_dropped -r 2 [ 1802.798562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 2 [ 1802.861698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_dropped -r 2 [ 1802.933244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_dropped -r 2 [ 1803.012555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_dropped -r 2 [ 1803.070339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_dropped -r 2 [ 1803.107573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1803.134346] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1803.499539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1803.518525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1803.537752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1803.572280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1803.610616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1803.658497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1803.690715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1803.713934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1803.732969] LNet: 88131:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1803.749981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1803.775865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1803.801932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1803.829334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1803.874444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1803.913645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1804.555541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_aborted -r 2 [ 1804.614264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 2 [ 1804.680558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_aborted -r 2 [ 1804.751230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 2 [ 1804.802528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_aborted -r 2 [ 1804.869493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_aborted -r 2 [ 1804.921141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_aborted -r 2 [ 1804.955973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_aborted -r 2 [ 1804.988553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1805.004226] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1805.265819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1805.297405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1805.337527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1805.367457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1805.408183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1805.452444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1805.489411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1805.545587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1805.565782] LNet: 88251:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1805.587316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1805.614971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1805.642592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1805.669171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1805.694506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1805.733029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1806.389697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_no_route -r 2 [ 1806.435857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 2 [ 1806.470477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_no_route -r 2 [ 1806.502678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 2 [ 1806.529626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_no_route -r 2 [ 1806.570181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_no_route -r 2 [ 1806.600238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_no_route -r 2 [ 1806.637876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_no_route -r 2 [ 1806.680258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1806.998573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1807.031834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1807.080616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1807.110493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1807.154558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1807.206975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1807.240906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1807.282428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1807.313504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1807.344946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1807.390248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1807.431171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1807.476900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1807.535367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1808.083569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e local_timeout -r 2 [ 1808.127829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 2 [ 1808.160239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e local_timeout -r 2 [ 1808.199371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 2 [ 1808.267609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e local_timeout -r 2 [ 1808.313402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e local_timeout -r 2 [ 1808.353201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e local_timeout -r 2 [ 1808.412199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e local_timeout -r 2 [ 1808.471442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1808.509500] LNet: There was an unexpected network error while writing to 192.168.201.151: rc = -22 [ 1808.513322] LNet: Skipped 1 previous similar message [ 1808.710859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1808.744780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1808.771661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1808.802437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1808.827346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1808.857384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1808.891133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1808.940542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1808.965417] LNet: 88492:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1808.974253] LNet: 88492:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1808.996347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1809.041631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1809.093556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1809.146764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1809.226740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1809.294669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1809.863555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp -m GET -e remote_dropped -r 2 [ 1809.892493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 2 [ 1809.916720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp -m GET -e remote_dropped -r 2 [ 1809.943180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 2 [ 1809.971123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp -m GET -e remote_dropped -r 2 [ 1810.003409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.151@tcp1 -m GET -e remote_dropped -r 2 [ 1810.034523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp -m GET -e remote_dropped -r 2 [ 1810.066454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.51@tcp1 -d 192.168.201.51@tcp1 -m GET -e remote_dropped -r 2 [ 1810.101587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.51@tcp 192.168.201.151@tcp [ 1810.122954] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 1810.365216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1810.401383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1810.440383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1810.490538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1810.548857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1810.600409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1810.649409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1810.707816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1810.766656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1810.827323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1810.898644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1810.932269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1810.977469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1811.016562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1817.360276] LNet: 88686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.371311] LNetError: 88686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.385383] LNet: Removed LNI 192.168.201.51@tcp [ 1817.434793] LNet: Removed LNI 192.168.201.51@tcp1 [ 1818.267569] Key type .llcrypt unregistered [ 1818.273817] Key type ._llcrypt unregistered [ 1836.274877] Key type ._llcrypt registered [ 1836.276810] Key type .llcrypt registered [ 1837.632972] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:17:56 (1784585876) [ 1838.203911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.218847] alg: No test for adler32 (adler32-zlib) [ 1839.079819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.122319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1839.134214] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1839.571847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1839.595583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.51@tcp [ 1839.605650] LNetError: 89669:0:(api-ni.c:4416:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1839.645694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.51@tcp [ 1839.693042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1846.057801] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:18:04 (1784585884) [ 1846.267548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1846.282709] LNet: 90259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1846.286238] LNetError: 90259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1846.300251] LNet: Removed LNI 192.168.201.51@tcp [ 1846.316558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.355703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1846.376598] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1847.146484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1847.158628] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 1848.378166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.51@tcp [ 1848.430749] LNetError: 90525:0:(api-ni.c:4416:lnet_ni_set_healthv()) manually adding local NI 192.168.201.51@tcp to recovery [ 1849.439644] LNet: 1 local NIs in recovery (showing 1): 192.168.201.51@tcp [ 1849.448780] LNet: There was an unexpected network error while writing to 192.168.201.51: rc = -22 [ 1849.774231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.51@tcp [ 1860.182306] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:18:18 (1784585898) [ 1862.606469] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg151-server [ 1864.971545] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 18:18:23 (1784585903) [ 1867.420358] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg151-server [ 1869.435170] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 18:18:27 (1784585907) [ 1871.449842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg151-server [ 1873.069179] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:18:31 (1784585911) [ 1873.684339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.735639] LNet: 92157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.746255] LNetError: 92157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.753110] LNet: Removed LNI 192.168.201.51@tcp [ 1873.788265] LNet: Removed LNI 192.168.201.51@tcp1 [ 1873.820037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.882396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1873.921048] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1874.828547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1874.922086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1874.986180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1875.091174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1875.149477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1875.183460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1883.974724] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:18:42 (1784585922) [ 1884.540620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.613419] LNet: 92883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.641472] LNetError: 92883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.656466] LNetError: 92883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1884.711375] LNet: Removed LNI 192.168.201.51@tcp [ 1884.782290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.850288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.876028] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1895.352509] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:18:53 (1784585933) [ 1895.471997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.493055] LNet: 93580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.496948] LNetError: 93580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.515246] LNet: Removed LNI 192.168.201.51@tcp [ 1895.548103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.688362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1895.732672] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1904.518678] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:19:02 (1784585942) [ 1906.239423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1908.099114] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:19:06 (1784585946) [ 1908.642495] LNet: 94622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.652968] LNetError: 94622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.667335] LNet: Removed LNI 192.168.201.51@tcp [ 1909.271112] Key type .llcrypt unregistered [ 1909.276190] Key type ._llcrypt unregistered [ 1915.848467] Key type ._llcrypt registered [ 1915.852519] Key type .llcrypt registered [ 1916.041224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1916.056627] alg: No test for adler32 (adler32-zlib) [ 1917.038145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1917.073725] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1925.167774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 1930.665580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.151@tcp [ 1931.073941] LNet: 95041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.090654] LNetError: 95041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.122621] LNet: Removed LNI 192.168.201.51@tcp [ 1931.649746] Key type .llcrypt unregistered [ 1931.651513] Key type ._llcrypt unregistered [ 1943.960702] Key type ._llcrypt registered [ 1943.963376] Key type .llcrypt registered [ 1945.241489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1946.853594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1948.577700] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:19:47 (1784585987) [ 1950.286674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1952.503703] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:19:50 (1784585990) [ 1954.413723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg151-server [ 1956.218054] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:19:54 (1784585994) [ 1958.343457] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg151-server [ 1960.252736] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:19:59 (1784585999) [ 1962.000664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1962.022543] alg: No test for adler32 (adler32-zlib) [ 1963.085868] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 1963.393431] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 1965.031432] Key type lgssc registered [ 1966.222703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1984.869957] Key type lgssc unregistered [ 1985.169073] LNet: 98215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.186663] LNetError: 98215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.199980] LNet: Removed LNI 192.168.201.51@tcp [ 1985.909885] Key type .llcrypt unregistered [ 1985.914963] Key type ._llcrypt unregistered [ 1999.411923] Key type ._llcrypt registered [ 1999.414495] Key type .llcrypt registered [ 2000.721392] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:20:39 (1784586039) [ 2001.215849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2001.232825] alg: No test for adler32 (adler32-zlib) [ 2002.120314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.169613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2002.199106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2002.246734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2002.290688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2008.547873] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:20:47 (1784586047) [ 2008.654931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2008.685907] LNet: 99636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.727907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.836212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2008.854867] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2009.790486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2009.811194] LNet: 99787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.816706] LNetError: 99787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.832605] LNet: Removed LNI 192.168.201.51@tcp [ 2009.853252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.900815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2009.948503] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2010.779894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2010.802677] LNetError: 99816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2010.817203] LNet: Removed LNI 192.168.201.51@tcp [ 2010.841287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.908763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2010.937257] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2019.382408] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:20:57 (1784586057) [ 2019.633489] LNet: 100403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.643045] LNet: 100403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2019.653897] LNetError: 100403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.673386] LNet: Removed LNI 192.168.201.51@tcp [ 2020.281989] Key type .llcrypt unregistered [ 2020.283726] Key type ._llcrypt unregistered [ 2027.090678] Key type ._llcrypt registered [ 2027.093622] Key type .llcrypt registered [ 2027.329270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2027.342869] alg: No test for adler32 (adler32-zlib) [ 2028.321991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.411466] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2028.416339] LNet: *** cfs_fail_loc=e003, val=100*** [ 2029.333489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2029.354666] LNet: *** cfs_fail_loc=e003, val=100*** [ 2029.358208] LNet: Skipped 2 previous similar messages [ 2029.363835] Ping reply received with mapping: [ 2029.366657] found metadata reply NI[192.168.201.51@tcp] [ 2029.369403] 33 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2029.372368] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2029.375479] 5a 5a 5a 5a 5a 5a 5a 5a [ 2029.491209] LNet: 100803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2029.497977] LNetError: 100803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.568703] LNet: Removed LNI 192.168.201.51@tcp [ 2031.297608] Key type .llcrypt unregistered [ 2031.300206] Key type ._llcrypt unregistered [ 2044.581493] Key type ._llcrypt registered [ 2044.583392] Key type .llcrypt registered [ 2045.777363] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:21:24 (1784586084) [ 2046.511980] Key type .llcrypt unregistered [ 2046.518947] Key type ._llcrypt unregistered [ 2053.390243] Key type ._llcrypt registered [ 2053.392316] Key type .llcrypt registered [ 2053.604657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2053.633704] alg: No test for adler32 (adler32-zlib) [ 2054.914808] LNetError: 101849:0:(api-ni.c:2793:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2055.659842] Key type .llcrypt unregistered [ 2055.664501] Key type ._llcrypt unregistered [ 2063.103570] Key type ._llcrypt registered [ 2063.119834] Key type .llcrypt registered [ 2064.852641] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:21:42 (1784586102) [ 2065.466583] Key type .llcrypt unregistered [ 2065.468218] Key type ._llcrypt unregistered [ 2073.495898] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2073.499112] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2076.014520] Key type ._llcrypt registered [ 2076.017640] Key type .llcrypt registered [ 2076.183335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2076.192876] alg: No test for adler32 (adler32-zlib) [ 2077.115988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.178404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2086.895879] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:22:05 (1784586125) [ 2087.140994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2087.167111] LNet: 103630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.178640] LNetError: 103630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2087.192653] LNet: Removed LNI 10.1.2.3@tcp [ 2087.211553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.114647] Key type .llcrypt unregistered [ 2088.118252] Key type ._llcrypt unregistered [ 2094.487173] Key type ._llcrypt registered [ 2094.489553] Key type .llcrypt registered [ 2094.649447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2094.658972] alg: No test for adler32 (adler32-zlib) [ 2095.620954] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2095.680628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2095.802646] LNet: 103919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2095.807454] LNetError: 103919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2096.877659] LNet: Removed LNI 192.168.201.51@tcp [ 2097.583514] Key type .llcrypt unregistered [ 2097.586255] Key type ._llcrypt unregistered [ 2111.152544] Key type ._llcrypt registered [ 2111.155311] Key type .llcrypt registered [ 2112.587492] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:22:30 (1784586150) [ 2113.343463] Key type .llcrypt unregistered [ 2113.345868] Key type ._llcrypt unregistered [ 2119.647888] Key type ._llcrypt registered [ 2119.653668] Key type .llcrypt registered [ 2119.970831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2119.992038] alg: No test for adler32 (adler32-zlib) [ 2129.614587] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:22:47 (1784586167) [ 2129.775783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.820351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.948685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2129.972773] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2130.794697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2130.833722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2130.864757] LNet: Removed LNI 192.168.201.51@tcp [ 2131.132651] LNet: 105665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.545623] Key type .llcrypt unregistered [ 2131.547896] Key type ._llcrypt unregistered [ 2145.927911] Key type ._llcrypt registered [ 2145.934443] Key type .llcrypt registered [ 2147.688379] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:23:05 (1784586185) [ 2148.134366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2148.156067] alg: No test for adler32 (adler32-zlib) [ 2149.097612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.162381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2149.197026] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2150.137810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2150.198100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.51@tcp [ 2150.237830] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2150.258529] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.262145] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 0 [ 2150.271945] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2150.274945] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2150.282148] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.286595] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 0 [ 2150.294795] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2150.296971] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2150.303466] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.307081] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 2200 [ 2150.313636] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2150.322371] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2150.325456] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2150.332333] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2150, 0, 2200 [ 2150.337196] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2157.771795] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:23:16 (1784586196) [ 2158.415547] LNet: 107207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.423547] LNetError: 107207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2159.474263] LNet: Removed LNI 192.168.201.51@tcp [ 2160.396716] Key type .llcrypt unregistered [ 2160.399567] Key type ._llcrypt unregistered [ 2167.869245] Key type ._llcrypt registered [ 2167.875319] Key type .llcrypt registered [ 2168.160794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2168.212253] alg: No test for adler32 (adler32-zlib) [ 2169.214239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2169.255108] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2177.670676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 2187.711559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2187.753148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2187.778595] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 2188.997447] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2189.195834] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2189.202695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2189.204628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2191.569163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2191.587391] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2203.260560] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:24:01 (1784586241) [ 2203.441389] LNet: 108710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.448121] LNetError: 108710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2203.455946] LNet: Removed LNI 192.168.201.51@tcp [ 2204.513093] LNet: Removed LNI 192.168.201.51@tcp1 [ 2204.518627] LNetError: 108710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2204.528323] LNetError: 108710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2204.552831] LNet: Removed LNI 10.1.2.3@tcp99 [ 2205.310741] Key type .llcrypt unregistered [ 2205.313488] Key type ._llcrypt unregistered [ 2212.037443] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2212.039615] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2214.723916] Key type ._llcrypt registered [ 2214.725969] Key type .llcrypt registered [ 2215.017241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2215.032793] alg: No test for adler32 (adler32-zlib) [ 2215.953921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.001413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2216.020952] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2216.964788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2217.003944] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2219.616811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2219.665881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2219.710377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2219.725601] LNetError: 109458:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2219.754699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2219.792122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2220.218996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2220.328325] LNet: 109479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.338197] LNetError: 109479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2220.345107] LNet: Removed LNI 10.1.2.3@tcp [ 2221.425261] LNet: Removed LNI 192.168.201.51@tcp [ 2222.157532] Key type .llcrypt unregistered [ 2222.160770] Key type ._llcrypt unregistered [ 2235.556758] Key type ._llcrypt registered [ 2235.574695] Key type .llcrypt registered [ 2237.447290] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:24:35 (1784586275) [ 2237.853977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2237.878543] alg: No test for adler32 (adler32-zlib) [ 2238.776938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.819494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2238.836618] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2239.400195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.51@tcp [ 2239.467304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg151-client@tcp [ 2246.219088] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:24:44 (1784586284) [ 2246.661533] LNet: 111013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2246.687447] LNetError: 111013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2246.725757] LNet: Removed LNI 192.168.201.51@tcp [ 2247.850946] Key type .llcrypt unregistered [ 2247.863309] Key type ._llcrypt unregistered [ 2256.448310] Key type ._llcrypt registered [ 2256.451315] Key type .llcrypt registered [ 2256.608974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2256.634400] alg: No test for adler32 (adler32-zlib) [ 2257.738367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2257.791214] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2257.852056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2265.686773] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:25:03 (1784586303) [ 2267.677200] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2269.336968] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:25:08 (1784586308) [ 2270.970683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2273.049469] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:25:11 (1784586311) [ 2273.164384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.225955] LNet: 112430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.244583] LNetError: 112430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2273.286035] LNet: Removed LNI 192.168.201.51@tcp [ 2273.310446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.363079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2273.384606] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2274.637575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2274.713295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2282.615954] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:25:21 (1784586321) [ 2282.728366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2282.759904] LNet: 113138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2282.769464] LNetError: 113138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2283.815751] LNet: Removed LNI 192.168.201.51@tcp [ 2283.856580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.909135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2283.966345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2284.190448] LNet: 113180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.589330] Key type .llcrypt unregistered [ 2284.591073] Key type ._llcrypt unregistered [ 2295.948348] Key type ._llcrypt registered [ 2295.950922] Key type .llcrypt registered [ 2297.520474] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:25:35 (1784586335) [ 2298.108961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2298.122671] alg: No test for adler32 (adler32-zlib) [ 2299.132142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.224636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2299.262214] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2300.441444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2300.524916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2307.775705] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:25:46 (1784586346) [ 2307.950368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2307.967444] LNet: 114700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.972616] LNetError: 114700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.038990] LNet: Removed LNI 192.168.201.51@tcp [ 2309.102335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.195374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2309.221408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2315.460691] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:25:54 (1784586354) [ 2315.580203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2315.596555] LNet: 115273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.614223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2315.681483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2316.091574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2316.132839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2316.189255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2316.662823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2316.703839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2316.784793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.807096] LNet: 115314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2316.826330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.872558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2317.238350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2317.289737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2317.369318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2317.748214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2317.778966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2324.127869] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:26:02 (1784586362) [ 2324.242383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2324.281466] LNet: 115908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2324.330145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2324.475700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2324.502724] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2325.498673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2325.518691] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2326.589889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2326.637952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2326.915499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2326.941441] LNetError: 116193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2326.952425] LNet: Removed LNI 192.168.201.51@tcp [ 2326.966853] LNet: Removed LNI 192.168.201.51@tcp2 [ 2327.002854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2327.105572] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2327.110689] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2327.116162] LNet: 116199:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2327.763469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2327.986804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.002224] LNetError: 116236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2328.007738] LNetError: 116236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2328.012115] LNet: Removed LNI 192.168.201.51@tcp [ 2328.017974] LNet: Removed LNI 192.168.201.51@tcp2 [ 2328.035805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.084666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2328.102757] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2328.977570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2329.002412] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2330.096961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2330.171263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2330.262579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2330.580397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2330.629620] LNet: 116521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2330.635056] LNet: 116521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2330.642127] LNetError: 116521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2330.653479] LNetError: 116521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2330.661543] LNet: Removed LNI 192.168.201.51@tcp [ 2331.696648] LNet: Removed LNI 192.168.201.51@tcp2 [ 2331.754481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.836025] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2331.840949] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2331.846269] LNet: 116527:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2332.265872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2332.466553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2332.484805] LNet: Removed LNI 192.168.201.51@tcp [ 2333.542835] LNet: Removed LNI 192.168.201.51@tcp2 [ 2333.622283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.698578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2333.737746] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2334.463620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2334.474827] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2335.318377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2335.388247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2335.679777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.724632] LNetError: 116844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2335.737269] LNetError: 116844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2335.748092] LNet: Removed LNI 192.168.201.51@tcp [ 2335.762770] LNet: Removed LNI 192.168.201.51@tcp2 [ 2335.783794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2335.847188] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2335.861746] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2335.870549] LNet: 116850:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2336.515930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2336.844420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.887171] LNet: Removed LNI 192.168.201.51@tcp [ 2337.903691] LNet: Removed LNI 192.168.201.51@tcp2 [ 2337.943204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.989983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2338.015916] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2338.635581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2338.664648] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2339.480705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2339.516103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2339.568525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2339.790312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.811474] LNet: 117171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.816196] LNet: 117171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2340.831260] LNet: Removed LNI 192.168.201.51@tcp [ 2341.877319] LNet: Removed LNI 192.168.201.51@tcp2 [ 2341.935659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.102154] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2342.108707] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2342.113599] LNet: 117178:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2342.905868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2343.177523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.199206] LNet: Removed LNI 192.168.201.51@tcp [ 2344.229725] LNet: Removed LNI 192.168.201.51@tcp2 [ 2344.284792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.346443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2344.373344] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2345.131339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2345.155581] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2346.238395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2346.301408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2346.387206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2346.841395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.860696] LNetError: 117512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2346.865673] LNetError: 117512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2346.871270] LNet: Removed LNI 192.168.201.51@tcp [ 2346.882423] LNet: Removed LNI 192.168.201.51@tcp2 [ 2346.904270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2346.953130] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2346.971760] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2346.987387] LNet: 117518:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.604893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2348.041464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.087156] LNet: Removed LNI 192.168.201.51@tcp [ 2349.094848] LNet: Removed LNI 192.168.201.51@tcp2 [ 2349.122221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2349.177512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2349.198512] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2349.807780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2349.826589] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2350.720347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2350.761679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2350.795739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2350.840747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2351.179344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.224094] LNet: Removed LNI 192.168.201.51@tcp [ 2353.254716] LNet: Removed LNI 192.168.201.51@tcp2 [ 2353.289959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.368394] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2353.371913] LNet: Added LNI 192.168.201.51@tcp2 [8/256/0/180] [ 2353.917559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2361.597967] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2363.249844] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:26:41 (1784586401) [ 2363.339828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2363.377230] LNet: 118631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2363.384701] LNet: 118631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2363.392835] LNetError: 118631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2363.398437] LNetError: 118631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2363.413600] LNet: Removed LNI 192.168.201.51@tcp [ 2364.458125] LNet: Removed LNI 192.168.201.51@tcp2 [ 2364.511964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.871690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2364.900084] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2365.663736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2365.695302] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2366.718074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2366.757836] LNet: Removed LNI 10.1.2.3@tcp [ 2366.771586] LNet: Removed LNI 192.168.201.51@tcp [ 2366.803147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.922661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2366.953572] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2366.959229] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2367.787478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2367.804867] LNet: Removed LNI 192.168.201.51@tcp [ 2367.819873] LNet: Removed LNI 10.1.2.3@tcp [ 2374.551689] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:26:52 (1784586412) [ 2374.678311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.760140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2374.908423] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2375.034095] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2375.043172] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2375.050085] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2377.495894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2377.535667] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2378.290762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2378.312762] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2380.183513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.51@tcp [ 2380.240253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2380.308488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.51@tcp -m PUT -e local_timeout -r 1 [ 2380.383862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2380.457603] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2380.463258] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2380.811101] LNetError: 119571:0:(lib-move.c:4766:lnet_parse()) 192.168.201.51@tcp, src 192.168.201.51@tcp: Dropping PUT (error -108 looking up sender) [ 2381.859707] LNet: Removed LNI 10.1.2.3@tcp [ 2382.897875] LNet: Removed LNI 192.168.201.51@tcp [ 2383.302699] Key type .llcrypt unregistered [ 2383.304936] Key type ._llcrypt unregistered [ 2397.302520] Key type ._llcrypt registered [ 2397.311494] Key type .llcrypt registered [ 2398.823476] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:27:17 (1784586437) [ 2399.401236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2399.425817] alg: No test for adler32 (adler32-zlib) [ 2400.432594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2400.784601] LNet: 120891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2401.630677] Key type .llcrypt unregistered [ 2401.633991] Key type ._llcrypt unregistered [ 2408.375137] Key type ._llcrypt registered [ 2408.378316] Key type .llcrypt registered [ 2408.722340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2408.779934] alg: No test for adler32 (adler32-zlib) [ 2409.821417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2409.893661] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2417.689195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 2417.918435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2418.618810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.151@tcp [ 2418.912420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2418.950866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2418.984367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2419.020362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2419.057236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2419.103342] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.151@tcp [ 2419.133420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2419.181231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2419.228964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2419.253709] LNet: 121340:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2419.272834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2419.298858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2419.324386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2419.399347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2419.467197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2419.531066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2420.378648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2420.586743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2420.634436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2420.694296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2420.736995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2420.793658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2420.852905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2420.907485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2420.955093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2420.992103] LNet: 121458:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2421.034811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2421.115654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2421.171411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2421.251708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2421.319290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2421.384138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2422.244927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.151@tcp [ 2422.567699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.600388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2422.643196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2422.681702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2422.712528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2422.738485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2422.769070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2422.794230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2422.813093] LNet: 121575:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.830760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2422.871426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2422.925099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2422.979845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2423.038808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2423.081756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2423.783989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2424.095553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2424.133445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2424.188117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2424.243898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2424.294744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2424.336331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2424.372836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2424.436777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2424.499406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2424.543790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2424.586873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2424.645078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2424.748127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2424.801279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2425.262435] LNet: 121738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.276018] LNetError: 121738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.298679] LNet: Removed LNI 192.168.201.51@tcp [ 2425.834606] Key type .llcrypt unregistered [ 2425.839198] Key type ._llcrypt unregistered [ 2447.153313] Key type ._llcrypt registered [ 2447.156388] Key type .llcrypt registered [ 2449.071682] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:28:07 (1784586487) [ 2449.764599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2449.816403] alg: No test for adler32 (adler32-zlib) [ 2450.830470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.538614] LNet: 122626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.903556] Key type .llcrypt unregistered [ 2452.907594] Key type ._llcrypt unregistered [ 2460.813315] Key type ._llcrypt registered [ 2460.823598] Key type .llcrypt registered [ 2461.110634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2461.138984] alg: No test for adler32 (adler32-zlib) [ 2462.127625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2462.156445] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2469.797598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.151@tcp [ 2479.605231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2479.666474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2479.697985] LNet: Added LNI 192.168.201.51@tcp1 [8/256/0/180] [ 2481.113744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2481.907350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.151@tcp [ 2482.177827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2482.205651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2482.245957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2482.274141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2482.301403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2482.327976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2482.351837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2482.379332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2482.409574] LNet: 123279:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2482.436856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2482.470970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2482.499807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2482.526819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2482.558103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2482.583593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2483.299199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2483.600821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2483.658720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2483.685300] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.151@tcp, 192.168.201.151@tcp1 [ 2483.738215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2483.805134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2483.835127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2483.877740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2483.910861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2483.957386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2484.002626] LNet: 123396:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2484.051304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2484.108233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2484.168147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2484.188766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2484.223488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2484.250984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2484.975155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.151@tcp [ 2485.278601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2485.295199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2485.314588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2485.343978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2485.377514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2485.408625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2485.446298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2485.496314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2485.515402] LNet: 123513:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2485.536794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2485.570230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2485.615660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2485.649830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2485.680874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2485.730339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2486.562165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2486.923136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2486.946913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2486.976107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2487.002409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2487.028158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2487.052824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2487.077705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2487.115730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2487.147431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2487.199650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2487.253778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2487.311972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2487.350462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2487.387302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2494.294801] LNet: 123711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2494.301272] LNetError: 123711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2495.327161] LNet: Removed LNI 192.168.201.51@tcp [ 2495.329747] LNetError: 123711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2495.346730] LNet: Removed LNI 192.168.201.51@tcp1 [ 2495.753581] Key type .llcrypt unregistered [ 2495.754840] Key type ._llcrypt unregistered [ 2517.129558] Key type ._llcrypt registered [ 2517.131503] Key type .llcrypt registered [ 2518.679149] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 18:29:16 (1784586556) [ 2520.632214] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2522.310846] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 18:29:21 (1784586561) [ 2531.480772] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 18:29:29 (1784586569) [ 2539.206325] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 18:29:37 (1784586577) [ 2546.840649] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 18:29:45 (1784586585) [ 2554.150206] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 18:29:52 (1784586592) [ 2562.177178] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 18:30:00 (1784586600) [ 2570.017456] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 18:30:08 (1784586608) [ 2578.460340] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 18:30:16 (1784586616) [ 2585.795396] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 18:30:24 (1784586624) [ 2593.752583] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 18:30:32 (1784586632) [ 2602.717944] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 18:30:40 (1784586640) [ 2610.371777] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 18:30:49 (1784586649) [ 2617.372811] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 18:30:56 (1784586656) [ 2625.186905] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 18:31:03 (1784586663) [ 2633.400231] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 18:31:12 (1784586672) [ 2642.169648] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 18:31:20 (1784586680) [ 2649.268174] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 18:31:27 (1784586687) [ 2656.599714] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 18:31:35 (1784586695) [ 2657.088861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2657.117308] alg: No test for adler32 (adler32-zlib) [ 2658.013328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2658.138880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2658.165177] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2659.611979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2659.655593] LNet: 134716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2659.664166] LNetError: 134716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2659.682580] LNet: Removed LNI 192.168.201.51@tcp [ 2659.716642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2659.806634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2659.827230] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2667.164877] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 18:31:45 (1784586705) [ 2674.953778] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 18:31:53 (1784586713) [ 2684.246823] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 18:32:02 (1784586722) [ 2693.181408] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 18:32:11 (1784586731) [ 2702.452155] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 18:32:20 (1784586740) [ 2711.900086] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 18:32:30 (1784586750) [ 2720.155993] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 18:32:38 (1784586758) [ 2727.192026] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 18:32:45 (1784586765) [ 2734.508308] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 18:32:53 (1784586773) [ 2741.988553] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 18:33:00 (1784586780) [ 2750.456836] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 18:33:09 (1784586789) [ 2757.845641] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 18:33:16 (1784586796) [ 2765.966311] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2528 sec ========== 18:33:24 (1784586804) [ 2766.393239] LNet: 142189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2766.401185] LNetError: 142189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2766.423439] LNet: Removed LNI 192.168.201.51@tcp [ 2767.286189] Key type .llcrypt unregistered [ 2767.289101] Key type ._llcrypt unregistered [ 2782.710225] Key type ._llcrypt registered [ 2782.744101] Key type .llcrypt registered [ 2783.325333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2783.346564] alg: No test for adler32 (adler32-zlib) [ 2784.525840] Lustre: Lustre: Build Version: 2.17.54_183_ga497ed6 [ 2784.926584] LNet: Added LNI 192.168.201.51@tcp [8/256/0/180] [ 2786.655778] Key type lgssc registered [ 2788.185564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2856.115600] Lustre: Mounted lustre-client [ 2860.694823] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2870.501119] Lustre: Unmounted lustre-client [ 2903.215874] Key type lgssc unregistered [ 2903.698752] LNet: 145119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2903.708972] LNetError: 145119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2903.738465] LNet: Removed LNI 192.168.201.51@tcp [ 2904.955285] Key type .llcrypt unregistered [ 2904.958839] Key type ._llcrypt unregistered