[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 512974937 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001017] APIC: Switch to symmetric I/O mode setup [ 0.002419] x2apic enabled [ 0.003017] Switched APIC routing to physical x2apic. [ 0.004021] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008027] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009024] pid_max: default: 32768 minimum: 301 [ 0.010173] LSM: Security Framework initializing [ 0.011074] Yama: becoming mindful. [ 0.012046] SELinux: Initializing. [ 0.013084] *** VALIDATE selinux *** [ 0.022154] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027267] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030152] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032110] *** VALIDATE tmpfs *** [ 0.034276] *** VALIDATE proc *** [ 0.035331] *** VALIDATE cgroup *** [ 0.036016] *** VALIDATE cgroup2 *** [ 0.037383] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038203] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039019] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040063] Spectre V2 : User space: Vulnerable [ 0.041021] Speculative Store Bypass: Vulnerable [ 0.044217] debug: unmapping init [mem 0xffffffff8da59000-0xffffffff8da60fff] [ 0.047000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047961] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049045] ... version: 2 [ 0.050025] ... bit width: 48 [ 0.051031] ... generic registers: 4 [ 0.052025] ... value mask: 0000ffffffffffff [ 0.053028] ... max period: 00007fffffffffff [ 0.054027] ... fixed-purpose events: 3 [ 0.055023] ... event mask: 000000070000000f [ 0.056422] rcu: Hierarchical SRCU implementation. [ 0.058823] smp: Bringing up secondary CPUs ... [ 0.059745] x86: Booting SMP configuration: [ 0.060049] .... node #0, CPUs: #1 #2 #3 [ 0.064186] smp: Brought up 1 node, 4 CPUs [ 0.066026] smpboot: Max logical packages: 1 [ 0.067031] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.257632] node 0 deferred pages initialised in 187ms [ 0.260255] devtmpfs: initialized [ 0.261296] x86/mm: Memory block size: 128MB [ 0.263568] gcov: version magic: 0x41383552 [ 0.267400] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.268126] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.269393] pinctrl core: initialized pinctrl subsystem [ 0.270282] [ 0.270774] ************************************************************* [ 0.271034] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.272031] ** ** [ 0.273092] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.274026] ** ** [ 0.275025] ** This means that this kernel is built to expose internal ** [ 0.276030] ** IOMMU data structures, which may compromise security on ** [ 0.277029] ** your system. ** [ 0.278027] ** ** [ 0.279023] ** If you see this message and you are not debugging the ** [ 0.280025] ** kernel, report this immediately to your vendor! ** [ 0.281025] ** ** [ 0.282026] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.283021] ************************************************************* [ 0.284841] NET: Registered protocol family 16 [ 0.287013] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.290080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.293080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.299146] cpuidle: using governor menu [ 0.301961] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.303587] PCI: Using configuration type 1 for base access [ 0.306118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.315152] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.318044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.321098] cryptd: max_cpu_qlen set to 1000 [ 0.324327] ACPI: Added _OSI(Module Device) [ 0.326020] ACPI: Added _OSI(Processor Device) [ 0.327028] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.329015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.334020] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.340577] ACPI: Interpreter enabled [ 0.342067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.343021] ACPI: Using IOAPIC for interrupt routing [ 0.346126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.349438] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.359941] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.362046] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.365036] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.370097] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.374361] acpiphp: Slot [2] registered [ 0.376151] acpiphp: Slot [5] registered [ 0.377110] acpiphp: Slot [6] registered [ 0.379167] acpiphp: Slot [3] registered [ 0.381137] acpiphp: Slot [4] registered [ 0.382131] acpiphp: Slot [7] registered [ 0.384254] acpiphp: Slot [8] registered [ 0.386154] acpiphp: Slot [9] registered [ 0.387133] acpiphp: Slot [10] registered [ 0.389144] acpiphp: Slot [11] registered [ 0.391164] acpiphp: Slot [12] registered [ 0.392209] acpiphp: Slot [13] registered [ 0.394130] acpiphp: Slot [14] registered [ 0.396151] acpiphp: Slot [15] registered [ 0.398145] acpiphp: Slot [16] registered [ 0.400154] acpiphp: Slot [17] registered [ 0.401146] acpiphp: Slot [18] registered [ 0.403193] acpiphp: Slot [19] registered [ 0.405178] acpiphp: Slot [20] registered [ 0.407130] acpiphp: Slot [21] registered [ 0.409226] acpiphp: Slot [22] registered [ 0.410164] acpiphp: Slot [23] registered [ 0.412136] acpiphp: Slot [24] registered [ 0.413141] acpiphp: Slot [25] registered [ 0.415185] acpiphp: Slot [26] registered [ 0.417142] acpiphp: Slot [27] registered [ 0.419224] acpiphp: Slot [28] registered [ 0.421095] acpiphp: Slot [29] registered [ 0.423147] acpiphp: Slot [30] registered [ 0.424150] acpiphp: Slot [31] registered [ 0.426076] PCI host bridge to bus 0000:00 [ 0.427021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.429025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.431030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.433032] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.435029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.438044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.440259] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.444220] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.447558] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.456021] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.460062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.464027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.466021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.469034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.472663] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.475794] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.478052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.481960] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.487015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.497999] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.505018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.512165] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.523024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.530019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.542021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.551073] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.563023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.579021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.649018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.662334] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.663345] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.665364] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.667305] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.669235] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.675147] iommu: Default domain type: Passthrough [ 0.676603] SCSI subsystem initialized [ 0.677218] ACPI: bus type USB registered [ 0.678154] usbcore: registered new interface driver usbfs [ 0.681184] usbcore: registered new interface driver hub [ 0.683149] usbcore: registered new device driver usb [ 0.686284] pps_core: LinuxPPS API ver. 1 registered [ 0.688016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.691101] PTP clock support registered [ 0.694009] EDAC MC: Ver: 3.0.0 [ 0.695212] PCI: Using ACPI for IRQ routing [ 0.698110] NetLabel: Initializing [ 0.699249] NetLabel: domain hash size = 128 [ 0.702016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.704092] NetLabel: unlabeled traffic allowed by default [ 0.706184] vgaarb: loaded [ 0.708387] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.710023] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.719317] clocksource: Switched to clocksource kvm-clock [ 0.838334] VFS: Disk quotas dquot_6.6.0 [ 0.840204] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.843136] *** VALIDATE ramfs *** [ 0.844557] *** VALIDATE hugetlbfs *** [ 0.846365] pnp: PnP ACPI init [ 0.849724] pnp: PnP ACPI: found 6 devices [ 0.878610] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.882573] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.885170] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.887359] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.892319] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.894714] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.897643] NET: Registered protocol family 2 [ 0.900441] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.905496] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.909330] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.914219] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.918062] TCP: Hash tables configured (established 65536 bind 65536) [ 0.921546] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.928027] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.931332] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.934522] NET: Registered protocol family 1 [ 0.938182] RPC: Registered named UNIX socket transport module. [ 0.940451] RPC: Registered udp transport module. [ 0.941863] RPC: Registered tcp transport module. [ 0.943434] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.945622] NET: Registered protocol family 44 [ 0.947042] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.948963] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.950669] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.952452] PCI: CLS 0 bytes, default 64 [ 0.954884] Unpacking initramfs... [ 2.405224] debug: unmapping init [mem 0xffff9e487cc64000-0xffff9e487ffcffff] [ 2.409370] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.411817] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.414960] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.922642] Initialise system trusted keyrings [ 2.924486] Key type blacklist registered [ 2.926403] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.936528] zbud: loaded [ 2.939664] *** VALIDATE nfs *** [ 2.940711] *** VALIDATE nfs4 *** [ 2.942189] pstore: using deflate compression [ 2.946494] Platform Keyring initialized [ 3.060176] NET: Registered protocol family 38 [ 3.062023] Key type asymmetric registered [ 3.064653] Asymmetric key parser 'x509' registered [ 3.067627] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.072364] io scheduler mq-deadline registered [ 3.074614] io scheduler kyber registered [ 3.077354] io scheduler bfq registered [ 3.080628] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.083982] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.086603] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.089097] ACPI: Power Button [PWRF] [ 3.095796] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.105765] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.126182] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.158674] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.187052] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.192785] Non-volatile memory driver v1.3 [ 3.194910] Linux agpgart interface v0.103 [ 3.235691] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.238994] vda: detected capacity change from 0 to 76681216 [ 3.258382] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.261601] vdb: detected capacity change from 0 to 1073741824 [ 3.270511] libphy: Fixed MDIO Bus: probed [ 3.290866] usbcore: registered new interface driver usbserial_generic [ 3.292909] usbserial: USB Serial support registered for generic [ 3.296719] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.299837] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.301109] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.302986] mousedev: PS/2 mouse device common for all mice [ 3.306570] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.307785] rtc_cmos 00:05: RTC can wake from S4 [ 3.312862] rtc_cmos 00:05: registered as rtc0 [ 3.314807] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.316743] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.318795] intel_pstate: CPU model not supported [ 3.327303] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.330700] hid: raw HID events driver (C) Jiri Kosina [ 3.334135] usbcore: registered new interface driver usbhid [ 3.336023] usbhid: USB HID core driver [ 3.337557] drop_monitor: Initializing network drop monitor service [ 3.339751] Initializing XFRM netlink socket [ 3.341726] NET: Registered protocol family 10 [ 3.346601] Segment Routing with IPv6 [ 3.348784] NET: Registered protocol family 17 [ 3.350849] mpls_gso: MPLS GSO support [ 3.357256] RAS: Correctable Errors collector initialized. [ 3.359595] AVX version of gcm_enc/dec engaged. [ 3.361063] AES CTR mode by8 optimization enabled [ 3.454450] sched_clock: Marking stable (3454407955, 0)->(4548396359, -1093988404) [ 3.458764] registered taskstats version 1 [ 3.461161] Loading compiled-in X.509 certificates [ 3.463947] zswap: loaded using pool lzo/zbud [ 3.493069] Key type big_key registered [ 3.508556] Key type encrypted registered [ 3.510160] ima: No TPM chip found, activating TPM-bypass! [ 3.512566] ima: Allocated hash algorithm: sha1 [ 3.514603] ima: No architecture policies found [ 3.516605] evm: Initialising EVM extended attributes: [ 3.518757] evm: security.selinux [ 3.520262] evm: security.ima [ 3.521354] evm: security.capability [ 3.522813] evm: HMAC attrs: 0x1 [ 3.525528] rtc_cmos 00:05: setting system clock to 2026-09-04 11:32:44 UTC (1788521564) [ 3.532381] debug: unmapping init [mem 0xffffffff8ea03000-0xffffffff8ebfffff] [ 3.537035] debug: unmapping init [mem 0xffffffff8d782000-0xffffffff8da58fff] [ 3.546150] Write protecting the kernel read-only data: 28672k [ 3.549713] debug: unmapping init [mem 0xffffffff8be03000-0xffffffff8bffffff] [ 3.552799] debug: unmapping init [mem 0xffffffff8c714000-0xffffffff8c7fffff] [ 3.591039] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.599839] systemd[1]: Detected virtualization kvm. [ 3.602084] systemd[1]: Detected architecture x86-64. [ 3.604193] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.632252] systemd[1]: No hostname configured. [ 3.648073] systemd[1]: Set hostname to . [ 3.655824] random: systemd: uninitialized urandom read (16 bytes read) [ 3.658614] systemd[1]: Initializing machine ID from random generator. [ 3.720364] random: ln: uninitialized urandom read (6 bytes read) [ 3.815277] random: systemd: uninitialized urandom read (16 bytes read) [ 3.817830] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.822683] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.827506] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.550631] device-mapper: uevent: version 1.0.3 [ 4.553569] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.354812] virtio_net virtio0 ens2: renamed from eth0 [ 5.375151] random: fast init done [ 5.408286] scsi host0: ata_piix [ 5.420372] scsi host1: ata_piix [ 5.427817] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.430846] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.321314] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.029167] random: crng init done [ 10.030701] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.591930] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.886966] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.174830] SELinux: Disabled at runtime. [ 12.240985] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.251298] systemd[1]: Detected virtualization kvm. [ 12.253648] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.824268] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.827840] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.832230] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.836384] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.840474] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.851175] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.857280] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ 12.921793] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 13.425240] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.812616] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.876193] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.999726] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.013329] EDAC sbridge: Ver: 1.1.2 [ 15.376735] Key type dns_resolver registered [ 15.703324] NFS: Registering the id_resolver key type [ 15.705481] Key type id_resolver registered [ 15.707302] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg412-client login: [ 42.435047] hrtimer: interrupt took 2774077 ns [ 81.884932] libcfs: loading out-of-tree module taints kernel. [ 82.014776] Key type ._llcrypt registered [ 82.038436] Key type .llcrypt registered [ 83.073534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 83.090737] alg: No test for adler32 (adler32-zlib) [ 84.684795] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 85.898350] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 87.663187] Key type lgssc registered [ 89.990676] Lustre: Echo OBD driver; http://www.lustre.org/ [ 232.753715] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2 [ 238.806454] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 255.936950] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing check_logdir /tmp/testlogs/ [ 258.527640] Lustre: lustre-OST0000-osc-ffff9e48c4dfd800: disconnect after 23s idle [ 261.417436] Lustre: DEBUG MARKER: oleg412-client.virtnet: executing yml_node [ 267.287720] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 269.961733] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 272.665041] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 274.760568] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Sep 4 07:37:14 EDT 2026 [ 295.601828] Lustre: Unmounted lustre-client [ 325.617280] Key type lgssc unregistered [ 325.910207] LNet: 8420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.917788] LNetError: 8420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 325.943360] LNet: Removed LNI 192.168.204.12@tcp [ 327.029220] Key type .llcrypt unregistered [ 327.036056] Key type ._llcrypt unregistered [ 354.274751] Key type ._llcrypt registered [ 354.276745] Key type .llcrypt registered [ 354.507433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 354.536675] alg: No test for adler32 (adler32-zlib) [ 355.553501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 355.631993] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 355.657493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 359.917625] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 360.085653] LNet: 9299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.099929] LNetError: 9299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 360.162804] LNet: Removed LNI 192.168.204.12@tcp [ 360.727488] Key type .llcrypt unregistered [ 360.728890] Key type ._llcrypt unregistered [ 366.830703] Key type ._llcrypt registered [ 366.832555] Key type .llcrypt registered [ 367.969751] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 07:38:47 (1788521927) [ 368.453612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 368.464488] alg: No test for adler32 (adler32-zlib) [ 369.368872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.440323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.454228] LNet: 9712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.478762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.569155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 369.596773] LNet: 9719:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 376.328353] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 07:38:56 (1788521936) [ 376.410585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.464428] LNet: 10288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.506535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.634135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 382.656331] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 07:39:02 (1788521942) [ 382.814315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.857927] LNet: 10882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.886877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.974598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 389.167875] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 07:39:08 (1788521948) [ 389.498653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.521591] LNet: 11467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.552092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.650762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 395.899704] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 07:39:15 (1788521955) [ 395.969661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.000460] LNet: 12045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.058962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.152373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 396.597039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 402.693823] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 07:39:22 (1788521962) [ 402.880387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.967871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.036445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 403.480830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 403.916574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 410.692320] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 07:39:30 (1788521970) [ 410.841323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.874957] LNet: 13232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.881675] LNet: 13232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 410.932874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.034830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 417.808911] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 07:39:37 (1788521977) [ 417.928249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.989355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.044470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 418.081757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 418.579936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 418.645631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 419.254797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 419.306490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 419.947763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 420.024519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 420.564957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 420.626912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 421.146817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 421.179303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 421.609551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 421.651571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 422.160355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 422.216267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 422.667465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 422.709730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 423.227628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 423.275532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 423.662513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 423.695355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 424.088503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 424.137120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 424.594752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 424.647789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 432.787860] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 07:39:52 (1788521992) [ 432.939030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.987424] LNet: 14503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.999679] LNet: 14503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 433.054450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.139966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 433.205227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 439.971792] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 07:39:59 (1788521999) [ 440.086723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.154568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.247907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 440.288165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 446.644917] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 07:40:06 (1788522006) [ 446.747724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.824294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.944624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 446.968813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 452.941034] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 07:40:12 (1788522012) [ 453.156901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.214697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.313177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 453.340326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 459.613400] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 07:40:19 (1788522019) [ 459.719428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.766713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.918184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 460.023135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 468.235693] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 07:40:27 (1788522027) [ 468.314134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.346565] LNet: 17430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.351616] LNet: 17430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 468.378897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.462654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 468.491571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 475.830689] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 07:40:35 (1788522035) [ 475.970511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.027827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.237815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.937253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 483.634508] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 07:40:43 (1788522043) [ 483.894255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.943983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.161275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 484.434896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 492.023646] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 07:40:51 (1788522051) [ 492.223846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.277298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.404022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.585060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.377192] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 07:40:59 (1788522059) [ 499.538977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.597674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.995094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 500.221903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 507.236586] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 07:41:06 (1788522066) [ 507.400244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.434473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.521650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.608763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 514.113306] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 07:41:13 (1788522073) [ 514.209755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.252153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.342612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 514.381575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 520.306712] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 07:41:20 (1788522080) [ 520.365100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.401542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.481292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 526.612420] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 07:41:26 (1788522086) [ 526.704940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.771301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.867897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 533.169690] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 07:41:32 (1788522092) [ 533.255043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.279530] LNet: 22865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.291833] LNet: 22865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 533.325130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.434656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 533.896151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 540.699286] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 07:41:40 (1788522100) [ 540.837619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.939963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.091193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 541.157862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 547.545746] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 07:41:47 (1788522107) [ 547.687167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.753177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.872210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 547.917216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 554.073991] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 07:41:53 (1788522113) [ 554.268103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.314279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.394551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 554.438827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 561.467036] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 07:42:00 (1788522120) [ 561.596891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.657144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.732761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 561.772560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 568.837363] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 07:42:08 (1788522128) [ 569.089819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.154490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.210968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 569.298640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 569.322976] LNetError: 25821:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 569.374353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 569.404949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 576.172965] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 07:42:15 (1788522135) [ 576.265971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.321449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.457057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XDRUeSZtBN [ 576.522308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XDRUeSZtBN [ 576.591328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.XDRUeSZtBN [ 583.142302] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 07:42:22 (1788522142) [ 583.355473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.408360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.482422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 583.526637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 583.581698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 589.643907] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 07:42:29 (1788522149) [ 590.101540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.145625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.229900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 590.283713] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 591.981746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 593.163410] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 595.095186] Key type lgssc registered [ 596.586699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 600.213457] LNetError: 27583:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.204.12@tcp99 has no available nets [ 602.669790] Key type lgssc unregistered [ 602.971532] LNetError: 28689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.986566] LNet: Removed LNI 192.168.204.12@tcp [ 603.562578] Key type .llcrypt unregistered [ 603.576186] Key type ._llcrypt unregistered [ 609.699278] Key type ._llcrypt registered [ 609.701487] Key type .llcrypt registered [ 611.001539] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 07:42:50 (1788522170) [ 611.733550] Key type .llcrypt unregistered [ 611.741112] Key type ._llcrypt unregistered [ 618.612282] Key type ._llcrypt registered [ 618.614351] Key type .llcrypt registered [ 618.802500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 618.827652] alg: No test for adler32 (adler32-zlib) [ 619.780366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.845513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 619.866284] LNet: 29599:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 619.873216] LNet: 29599:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 619.877857] LNet: 29599:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 619.895378] LNet: Message forwarding will be enabled in 2s [ 623.008604] LNet: Message forwarding enabled [ 623.446348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 623.471798] LNet: Message forwarding will stop in 2s to 6s [ 627.116034] LNet: Message forwarding disabled [ 628.086823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 628.104230] LNet: 29699:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 628.110453] LNet: 29699:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 628.116170] LNet: 29699:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 628.125627] LNet: Message forwarding will be enabled in 2s [ 628.274580] LNet: 29712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.916505] Key type .llcrypt unregistered [ 628.918750] Key type ._llcrypt unregistered [ 641.664451] Key type ._llcrypt registered [ 641.667961] Key type .llcrypt registered [ 643.098487] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 07:43:22 (1788522202) [ 643.694898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 643.712023] alg: No test for adler32 (adler32-zlib) [ 644.664543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.790354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 650.952851] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 07:43:30 (1788522210) [ 651.210698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.230569] LNet: 31112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.251995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.311763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 651.342876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 651.381629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 651.415497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 651.452453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 651.493862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 651.522565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 651.549720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 651.598892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 651.630422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 651.659941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 651.701457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 651.744486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 651.800781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 651.833952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 651.868988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 651.900758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 651.936611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 651.970771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 651.991254] LNetError: 31155:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 658.650242] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 07:43:38 (1788522218) [ 658.780693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.819905] LNet: 31720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.862764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.972549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 667.842490] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 07:43:46 (1788522226) [ 670.000923] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg412-server [ 672.262335] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 07:43:51 (1788522231) [ 672.571361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.609160] LNet: 32657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.646044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.717260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 672.763103] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 674.053514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[13-17]@tcp [ 674.074402] LNet: 32790:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 674.155594] LNetError: 32661:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.16@tcp is being used as a gateway but routing feature is not turned on [ 674.177889] LNetError: 32661:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.16@tcp has gone from up to down [ 681.426407] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 07:44:01 (1788522241) [ 683.581077] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg412-server [ 685.067496] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 07:44:04 (1788522244) [ 685.205542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.242777] LNet: 33727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.252581] LNetError: 33727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.284722] LNet: Removed LNI 192.168.204.12@tcp [ 685.304720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.358588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.379649] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 686.114763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[13-18/2]@tcp [ 686.146958] LNet: 33845:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 686.156276] LNet: 33845:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 686.172507] LNetError: 33732:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.13@tcp is being used as a gateway but routing feature is not turned on [ 686.176429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[13-18/2]@tcp [ 686.187495] LNetError: 33732:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 686.187523] LNetError: 33732:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.13@tcp has gone from up to down [ 686.187528] LNetError: 33732:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 692.803891] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 07:44:12 (1788522252) [ 692.993737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.041232] LNet: 34436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.085733] LNetError: 34436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.146208] LNet: Removed LNI 192.168.204.12@tcp [ 693.203696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.320871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 693.377581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.414346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.570796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 693.671538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.730857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.806920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 693.930642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.983156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.077826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.241340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.302142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.358516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 694.451263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.493629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.551905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 694.656952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.695464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.744811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 694.815826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.870417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.974731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 695.079195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.118723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.195184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 695.318573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.398379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.499304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 695.636765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.702092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.823096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 695.884551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.954152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.040802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 702.085884] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 07:44:21 (1788522261) [ 704.326986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg412-server [ 706.336989] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 07:44:25 (1788522265) [ 708.365427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg412-server [ 710.042612] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 07:44:29 (1788522269) [ 710.343836] LNet: 35842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.351246] LNet: 35842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 710.992167] Key type .llcrypt unregistered [ 710.995321] Key type ._llcrypt unregistered [ 717.524543] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 717.607345] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 717.613409] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 717.620820] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 720.196600] Key type ._llcrypt registered [ 720.203370] Key type .llcrypt registered [ 720.464231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 720.478607] alg: No test for adler32 (adler32-zlib) [ 721.477365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.611331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 721.653898] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 722.781876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 722.816897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 725.789800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 725.828791] LNet: Removed LNI 10.1.2.3@tcp [ 726.021321] LNet: 36559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.037749] LNetError: 36559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.061919] LNet: Removed LNI 192.168.204.12@tcp [ 726.770887] Key type .llcrypt unregistered [ 726.778775] Key type ._llcrypt unregistered [ 741.045376] Key type ._llcrypt registered [ 741.047075] Key type .llcrypt registered [ 742.549743] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 07:45:01 (1788522301) [ 743.324546] Key type .llcrypt unregistered [ 743.330322] Key type ._llcrypt unregistered [ 749.939469] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 750.089207] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 750.095665] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 750.100621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 752.898510] Key type ._llcrypt registered [ 752.900752] Key type .llcrypt registered [ 753.163396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.194160] alg: No test for adler32 (adler32-zlib) [ 754.098511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.172269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.195848] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 754.958370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 754.975959] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 757.925974] LNet: 38044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.930986] LNetError: 38044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 757.935295] LNet: Removed LNI 10.1.2.3@tcp [ 757.943982] LNet: Removed LNI 192.168.204.12@tcp [ 758.847668] Key type .llcrypt unregistered [ 758.853266] Key type ._llcrypt unregistered [ 773.517246] Key type ._llcrypt registered [ 773.523862] Key type .llcrypt registered [ 775.530589] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 07:45:34 (1788522334) [ 776.064298] Key type .llcrypt unregistered [ 776.065927] Key type ._llcrypt unregistered [ 784.825902] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 784.966320] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 784.979121] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 784.987740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 787.917767] Key type ._llcrypt registered [ 787.919881] Key type .llcrypt registered [ 788.163743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.192304] alg: No test for adler32 (adler32-zlib) [ 789.285723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.335869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 789.363754] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 792.018872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 792.040723] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.833560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 794.855290] LNet: Removed LNI 10.1.2.3@tcp [ 794.870036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 794.914844] LNet: Removed LNI 10.1.2.31@tcp [ 794.936584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 794.962341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 795.815153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 795.853360] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 797.036835] LNet: 39962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.048859] LNetError: 39962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 797.060739] LNet: Removed LNI 10.1.2.31@tcp [ 797.089887] LNet: Removed LNI 10.1.2.3@tcp [ 797.869704] Key type .llcrypt unregistered [ 797.872483] Key type ._llcrypt unregistered [ 809.924644] Key type ._llcrypt registered [ 809.930241] Key type .llcrypt registered [ 811.157390] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 07:46:10 (1788522370) [ 811.470754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 811.483786] alg: No test for adler32 (adler32-zlib) [ 812.386692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 812.435496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 812.470440] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 813.166412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.187810] LNet: 40912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.196674] LNetError: 40912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.218191] LNet: Removed LNI 192.168.204.12@tcp [ 813.245728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.346944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.12@tcp [ 813.378557] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 814.161629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.12@tcp [ 814.197489] LNet: Removed LNI 192.168.204.12@tcp [ 814.258428] LNet: 41047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.051417] Key type .llcrypt unregistered [ 815.056679] Key type ._llcrypt unregistered [ 824.624571] Key type ._llcrypt registered [ 824.627250] Key type .llcrypt registered [ 825.994535] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 07:46:25 (1788522385) [ 827.902594] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg412-server [ 830.026073] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 07:46:29 (1788522389) [ 830.452265] Key type .llcrypt unregistered [ 830.454080] Key type ._llcrypt unregistered [ 837.077513] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 837.284560] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 837.295081] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 837.303526] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 839.923267] Key type ._llcrypt registered [ 839.929383] Key type .llcrypt registered [ 840.155593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.185941] alg: No test for adler32 (adler32-zlib) [ 841.169223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.243875] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 842.040937] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 844.840672] LNet: 42875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.871044] LNetError: 42875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.881617] LNet: Removed LNI 192.168.204.12@tcp1 [ 844.904038] LNet: Removed LNI 10.1.2.3@tcp2 [ 845.684469] Key type .llcrypt unregistered [ 845.688089] Key type ._llcrypt unregistered [ 858.112824] Key type ._llcrypt registered [ 858.114809] Key type .llcrypt registered [ 859.368178] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 07:46:58 (1788522418) [ 859.731700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.750731] alg: No test for adler32 (adler32-zlib) [ 860.641592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.687049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 860.710564] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 861.393709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.413367] LNet: 43842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.420199] LNetError: 43842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.440536] LNet: Removed LNI 192.168.204.12@tcp [ 861.455627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.519602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 861.539671] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 867.112668] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 07:47:06 (1788522426) [ 867.235142] LNet: 44429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.241529] LNetError: 44429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.264072] LNet: Removed LNI 192.168.204.12@tcp [ 868.014421] Key type .llcrypt unregistered [ 868.022114] Key type ._llcrypt unregistered [ 874.171672] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 874.416261] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 874.425320] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 874.430361] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 877.189849] Key type ._llcrypt registered [ 877.191930] Key type .llcrypt registered [ 877.452381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 877.488468] alg: No test for adler32 (adler32-zlib) [ 878.403976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.459927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 878.488547] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 879.094370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 879.119241] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 881.210566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.228883] LNet: 45174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.236808] LNetError: 45174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.244987] LNet: Removed LNI 10.1.2.3@tcp [ 881.254081] LNet: Removed LNI 192.168.204.12@tcp [ 881.274666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.334932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 881.364981] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 881.372357] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 881.883589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.904640] LNet: 45201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.909781] LNetError: 45201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.914766] LNetError: 45201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 881.920596] LNet: Removed LNI 10.1.2.3@tcp [ 881.933747] LNet: Removed LNI 192.168.204.12@tcp [ 881.994415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.077448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 882.119693] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 882.128332] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 882.608018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 882.646444] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 883.540103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.573428] LNet: 45364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.588193] LNetError: 45364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.602545] LNetError: 45364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 883.612334] LNet: Removed LNI 10.1.2.3@tcp [ 883.617368] LNet: Removed LNI 192.168.204.12@tcp [ 883.630729] LNet: Removed LNI 192.168.204.12@tcp2 [ 883.682812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.763332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 883.785522] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 883.798474] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.804294] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 891.224782] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 07:47:30 (1788522450) [ 891.323305] LNet: 45952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 891.342688] LNetError: 45952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 891.350833] LNetError: 45952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 891.361883] LNet: Removed LNI 10.1.2.3@tcp [ 891.365056] LNet: Removed LNI 192.168.204.12@tcp [ 891.386273] LNet: Removed LNI 192.168.204.12@tcp2 [ 892.062757] Key type .llcrypt unregistered [ 892.064730] Key type ._llcrypt unregistered [ 897.586086] Key type ._llcrypt registered [ 897.590993] Key type .llcrypt registered [ 897.807279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 897.835843] alg: No test for adler32 (adler32-zlib) [ 898.705831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.752876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 898.779570] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 899.625726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.654866] LNet: 46353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.665741] LNetError: 46353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.689206] LNet: Removed LNI 192.168.204.12@tcp [ 899.711966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.761822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 899.788321] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 900.175486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.194193] LNet: 46383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.201172] LNetError: 46383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.216259] LNet: Removed LNI 192.168.204.12@tcp [ 900.235680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.288693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 900.317375] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 900.663666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 900.691324] LNet: Removed LNI 192.168.204.12@tcp [ 900.715074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.785836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 900.808617] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 901.156647] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 901.287511] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 901.294277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 901.299074] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 903.662165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.683198] LNet: 46494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.688318] LNet: 46494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 903.693483] LNetError: 46494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.699665] LNetError: 46494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 903.717682] LNet: Removed LNI 192.168.204.12@tcp [ 903.733109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.808137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 903.831743] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 909.944178] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 07:47:49 (1788522469) [ 910.163323] LNet: 47091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.169249] LNetError: 47091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.190838] LNet: Removed LNI 192.168.204.12@tcp [ 910.937739] Key type .llcrypt unregistered [ 910.941395] Key type ._llcrypt unregistered [ 916.754698] Key type ._llcrypt registered [ 916.759955] Key type .llcrypt registered [ 916.997187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.027504] alg: No test for adler32 (adler32-zlib) [ 917.932721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.002468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 918.036835] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 918.794516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.808723] LNet: 47486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.813603] LNetError: 47486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.828779] LNet: Removed LNI 192.168.204.12@tcp [ 918.857204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.945752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 918.980435] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 925.501753] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 07:48:05 (1788522485) [ 925.667981] LNet: 48075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.680308] LNetError: 48075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.735732] LNet: Removed LNI 192.168.204.12@tcp [ 926.418488] Key type .llcrypt unregistered [ 926.428155] Key type ._llcrypt unregistered [ 932.209130] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 932.302764] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 932.308800] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 932.313430] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 934.910057] Key type ._llcrypt registered [ 934.913313] Key type .llcrypt registered [ 935.085420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 935.106715] alg: No test for adler32 (adler32-zlib) [ 936.028906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.090417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 936.113101] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 936.991903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.009325] LNet: 48527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.017123] LNetError: 48527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.036468] LNet: Removed LNI 192.168.204.12@tcp [ 937.053150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.134172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 937.153897] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 942.877393] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 07:48:22 (1788522502) [ 943.179384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.199610] LNet: 49125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.205635] LNetError: 49125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.220511] LNet: Removed LNI 192.168.204.12@tcp [ 943.240728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.371739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 948.860817] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 07:48:28 (1788522508) [ 948.994848] LNet: 49702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.539563] Key type .llcrypt unregistered [ 949.541620] Key type ._llcrypt unregistered [ 956.181482] Key type ._llcrypt registered [ 956.186654] Key type .llcrypt registered [ 956.383492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.392837] alg: No test for adler32 (adler32-zlib) [ 957.438595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.522238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 957.549760] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 958.495431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.538544] LNet: 50102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.548978] LNetError: 50102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.588545] LNet: Removed LNI 192.168.204.12@tcp [ 958.612948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.691730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 958.721731] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 965.029825] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 07:48:44 (1788522524) [ 965.242507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.283216] LNet: 50695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.292799] LNetError: 50695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.315044] LNet: Removed LNI 192.168.204.12@tcp [ 965.332954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.395358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 971.312236] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 07:48:50 (1788522530) [ 972.679994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 974.230237] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 07:48:53 (1788522533) [ 975.562343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 977.194107] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 07:48:56 (1788522536) [ 978.739915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 980.286274] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 07:48:59 (1788522539) [ 981.814759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 983.647265] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 07:49:03 (1788522543) [ 985.464515] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 987.166416] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 07:49:06 (1788522546) [ 987.268295] LNet: 52739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.890964] Key type .llcrypt unregistered [ 987.895904] Key type ._llcrypt unregistered [ 994.390139] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 994.503993] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 994.518027] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 994.533151] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 997.194323] Key type ._llcrypt registered [ 997.197652] Key type .llcrypt registered [ 997.391979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.405905] alg: No test for adler32 (adler32-zlib) [ 998.379735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.449248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.470884] LNet: Added LNI 192.168.204.12@tcp [8/512/0/180] [ 998.477164] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 998.478720] LNet: 53065:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 998.599668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.620657] LNet: 53084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.625827] LNetError: 53084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 998.630593] LNet: Removed LNI 10.1.2.3@tcp [ 998.632365] LNetError: 53084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.643780] LNet: Removed LNI 192.168.204.12@tcp [ 998.689663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.833218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 998.863651] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 998.952240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.970835] LNet: 53108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.978305] LNetError: 53108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.000922] LNet: Removed LNI 192.168.204.12@tcp [ 999.025301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.097226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 999.146627] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 999.157592] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 999.160728] LNet: 53116:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 999.304942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.329921] LNet: 53134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.338344] LNetError: 53134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 999.346380] LNet: Removed LNI 10.1.2.3@tcp [ 999.351605] LNetError: 53134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.368236] LNet: Removed LNI 192.168.204.12@tcp [ 999.406029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.501804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 999.539534] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 999.547402] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 999.549626] LNet: 53142:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 999.761985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.785423] LNet: 53168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.790510] LNetError: 53168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 999.795838] LNet: Removed LNI 10.1.2.3@tcp [ 999.797510] LNetError: 53168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.809622] LNet: Removed LNI 192.168.204.12@tcp [ 999.828854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.893947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 999.922622] LNet: Added LNI 192.168.204.12@tcp [8/256/0/182] [ 999.927940] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1000.009146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1000.026396] LNet: 53196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.034170] LNetError: 53196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1000.040781] LNet: Removed LNI 10.1.2.3@tcp [ 1000.043482] LNetError: 53196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.057800] LNet: Removed LNI 192.168.204.12@tcp [ 1000.076194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.151849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1000.186215] LNet: Added LNI 192.168.204.12@tcp [8/256/0/182] [ 1000.190801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1000.192741] LNet: 53204:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1000.258958] LNet: Removed LNI 10.1.2.3@tcp [ 1000.268796] LNet: Removed LNI 192.168.204.12@tcp [ 1001.032996] Key type .llcrypt unregistered [ 1001.039375] Key type ._llcrypt unregistered [ 1012.296524] Key type ._llcrypt registered [ 1012.298434] Key type .llcrypt registered [ 1013.385627] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 07:49:33 (1788522573) [ 1015.214296] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1016.902570] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 07:49:36 (1788522576) [ 1017.350623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1017.370511] alg: No test for adler32 (adler32-zlib) [ 1018.311975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.384145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1018.437613] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1019.246459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.12 [ 1019.281042] LNet: 54467:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1019.297844] LNet: Removed LNI 192.168.204.12@tcp [ 1025.675737] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 07:49:45 (1788522585) [ 1026.076931] LNet: 55037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.088469] LNetError: 55037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.115553] LNet: Removed LNI 192.168.204.12@tcp [ 1026.708429] Key type .llcrypt unregistered [ 1026.711398] Key type ._llcrypt unregistered [ 1033.855332] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1034.039716] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1035.780742] Key type ._llcrypt registered [ 1035.783358] Key type .llcrypt registered [ 1035.964236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.988170] alg: No test for adler32 (adler32-zlib) [ 1037.449406] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1043.780377] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 07:50:03 (1788522603) [ 1043.880811] LNet: 55930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.888127] LNetError: 55930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1043.922695] LNet: Removed LNI 10.1.2.3@tcp [ 1044.600957] Key type .llcrypt unregistered [ 1044.610092] Key type ._llcrypt unregistered [ 1050.498899] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1050.731624] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1050.735191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1050.739236] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1053.477695] Key type ._llcrypt registered [ 1053.480398] Key type .llcrypt registered [ 1053.671702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1053.730700] alg: No test for adler32 (adler32-zlib) [ 1054.648698] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1060.931509] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 07:50:20 (1788522620) [ 1062.473363] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1064.471728] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 07:50:23 (1788522623) [ 1064.689670] LNet: 57119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.698645] LNetError: 57119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1064.721782] LNet: Removed LNI 10.1.2.3@tcp [ 1065.341546] Key type .llcrypt unregistered [ 1065.345637] Key type ._llcrypt unregistered [ 1072.031552] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1072.236633] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1072.240797] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1072.245804] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1075.150198] Key type ._llcrypt registered [ 1075.158438] Key type .llcrypt registered [ 1075.465860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1075.491887] alg: No test for adler32 (adler32-zlib) [ 1076.450971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1076.582194] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1085.897351] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 07:50:45 (1788522645) [ 1086.746884] LNet: 58346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.752935] LNetError: 58346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1086.764784] LNet: Removed LNI 10.1.2.3@tcp [ 1087.583588] Key type .llcrypt unregistered [ 1087.587651] Key type ._llcrypt unregistered [ 1094.134427] Key type ._llcrypt registered [ 1094.136242] Key type .llcrypt registered [ 1094.397802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1094.418457] alg: No test for adler32 (adler32-zlib) [ 1095.340549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1095.369787] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1102.524899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1103.211595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_interrupt -r 1 [ 1103.243340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_interrupt -r 1 [ 1103.273360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1103.289301] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1103.476853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1103.511840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1103.565408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1103.583870] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1103.611584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1103.656673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1103.690924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1103.742530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1103.795925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1103.829397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1103.856400] LNet: 58783:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1103.881689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1103.939164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.000841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.049787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.110229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.142877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.177289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1104.839093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_dropped -r 1 [ 1104.898565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_dropped -r 1 [ 1104.933924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1104.955426] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1105.234259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1105.296402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1105.338657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1105.407668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1105.446643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1105.490709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1105.552662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1105.589955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1105.641430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1105.655835] LNet: 58894:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.672921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1105.714935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.777714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.826796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.874905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.943467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.997046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1106.663766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_aborted -r 1 [ 1106.697036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_aborted -r 1 [ 1106.761816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1106.783133] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1106.792548] LNet: Skipped 1 previous similar message [ 1107.059074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.082860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.110802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.138314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.172846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1107.198890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.247793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.292816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.331186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.353372] LNet: 59005:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.395698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.436493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.475821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.518899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.566226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.604920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.650542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.260654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_no_route -r 1 [ 1108.287309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_no_route -r 1 [ 1108.314041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1108.545409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.577957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.602856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.629422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.655932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1108.696759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.727856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.760383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.795409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.833791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.867932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.893135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.921926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.962493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.992862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.033209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1109.540232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_timeout -r 1 [ 1109.583052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_timeout -r 1 [ 1109.633406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1109.653437] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1109.659236] LNet: Skipped 1 previous similar message [ 1109.945527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.974308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.002222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.032067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.067286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1110.092504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.130781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.158208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.179943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.194973] LNet: 59228:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.201197] LNet: 59228:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1110.215288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.237623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.270941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.296930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.354305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.399563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.439593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1111.021847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_error -r 1 [ 1111.065590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_error -r 1 [ 1111.109892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1111.420902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.461074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1111.518960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.570475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.628329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1111.658464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.719607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.790355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.823464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.855173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.898509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.955903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.014464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.077486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.112066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.151498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1112.439486] LNet: 59380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.449329] LNetError: 59380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.463810] LNet: Removed LNI 192.168.204.12@tcp [ 1112.951830] Key type .llcrypt unregistered [ 1112.953640] Key type ._llcrypt unregistered [ 1130.309450] Key type ._llcrypt registered [ 1130.311816] Key type .llcrypt registered [ 1131.752708] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 07:51:31 (1788522691) [ 1133.057808] Key type .llcrypt unregistered [ 1133.060715] Key type ._llcrypt unregistered [ 1139.511510] Key type ._llcrypt registered [ 1139.514922] Key type .llcrypt registered [ 1139.707327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.722367] alg: No test for adler32 (adler32-zlib) [ 1140.685788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1140.735842] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1148.680071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1159.355964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1159.416676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1159.456073] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1161.227973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_interrupt -r 1 [ 1161.287480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_interrupt -r 1 [ 1161.338413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_interrupt -r 1 [ 1161.368307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_interrupt -r 1 [ 1161.415775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_interrupt -r 1 [ 1161.464925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_interrupt -r 1 [ 1161.515690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_interrupt -r 1 [ 1161.590073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_interrupt -r 1 [ 1161.658392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1161.698322] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1161.706619] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1161.966419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1162.000417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1162.044239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1162.094436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1162.129194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1162.177159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1162.201449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1162.233186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1162.258736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1162.282282] LNet: 60844:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1162.296250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1162.320659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1162.350156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1162.378478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1162.410634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.442754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.472592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1163.044897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_dropped -r 1 [ 1163.083862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_dropped -r 1 [ 1163.135136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_dropped -r 1 [ 1163.189791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_dropped -r 1 [ 1163.245119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_dropped -r 1 [ 1163.300560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_dropped -r 1 [ 1163.343445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_dropped -r 1 [ 1163.376165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_dropped -r 1 [ 1163.413782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1163.432906] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1163.439580] LNet: Skipped 2 previous similar messages [ 1163.608597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.636427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.662264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.687940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.727345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1163.756955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.790989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.836379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.886844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.923828] LNet: 60967:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.949826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.987595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1164.026960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.094651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1164.142301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.195899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.269070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1164.874990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_aborted -r 1 [ 1164.918097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_aborted -r 1 [ 1164.971604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_aborted -r 1 [ 1165.034822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_aborted -r 1 [ 1165.097198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_aborted -r 1 [ 1165.130814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_aborted -r 1 [ 1165.173783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_aborted -r 1 [ 1165.201259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_aborted -r 1 [ 1165.235077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1165.251903] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1165.257402] LNet: Skipped 2 previous similar messages [ 1165.486426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.536815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.560382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.594181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.619075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1165.654775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.693885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.727083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.756579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.779550] LNet: 61090:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.805117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.854100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.900623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.931840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.985749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.030430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.070271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1166.529704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_no_route -r 1 [ 1166.564445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_no_route -r 1 [ 1166.592258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_no_route -r 1 [ 1166.629801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_no_route -r 1 [ 1166.655215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_no_route -r 1 [ 1166.681458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_no_route -r 1 [ 1166.719851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_no_route -r 1 [ 1166.766945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_no_route -r 1 [ 1166.813215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1167.036779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.066887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.106296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.157374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.192953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1167.223797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.261197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.293311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.316603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.347380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.379817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.408790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.430523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.483517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.524552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.548898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.962401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_timeout -r 1 [ 1167.992278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_timeout -r 1 [ 1168.033856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_timeout -r 1 [ 1168.069642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_timeout -r 1 [ 1168.097030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_timeout -r 1 [ 1168.148737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_timeout -r 1 [ 1168.190050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_timeout -r 1 [ 1168.245361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_timeout -r 1 [ 1168.296675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1168.327317] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1168.332091] LNet: Skipped 5 previous similar messages [ 1168.572886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.606952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.656470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.696819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.737187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1168.792650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.827794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.892446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.945133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.967575] LNet: 61337:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.978912] LNet: 61337:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1169.027641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.090807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.116360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.141876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.165715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.220362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.264226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.724444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_error -r 1 [ 1169.789928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_error -r 1 [ 1169.850485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_error -r 1 [ 1169.907683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_error -r 1 [ 1169.948373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_error -r 1 [ 1169.991129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_error -r 1 [ 1170.026849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_error -r 1 [ 1170.066924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_error -r 1 [ 1170.100701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1170.325798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.354449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.377873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.398799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.421794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1170.443022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.468804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.494177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.519516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.540588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.567832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.588987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.613668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.647951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.684526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.709627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.945880] LNet: 61536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1175.951958] LNetError: 61536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.960366] LNet: Removed LNI 192.168.204.12@tcp [ 1175.969904] LNet: Removed LNI 192.168.204.12@tcp1 [ 1176.302712] Key type .llcrypt unregistered [ 1176.305639] Key type ._llcrypt unregistered [ 1191.788498] Key type ._llcrypt registered [ 1191.793740] Key type .llcrypt registered [ 1192.778444] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 07:52:32 (1788522752) [ 1193.474984] Key type .llcrypt unregistered [ 1193.476819] Key type ._llcrypt unregistered [ 1199.179680] Key type ._llcrypt registered [ 1199.184459] Key type .llcrypt registered [ 1199.389564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1199.418492] alg: No test for adler32 (adler32-zlib) [ 1200.349536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1200.371873] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1206.983666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1207.786664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_dropped -r 1 [ 1207.827379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_dropped -r 1 [ 1207.862950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1207.880327] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1208.121994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1208.167209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1208.192076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1208.213819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1208.238192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1208.262584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1208.285875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1208.324450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1208.357438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1208.385763] LNet: 62781:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1208.417841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1208.477611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1208.506573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1208.533891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1208.561326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1208.588499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1208.624028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1209.099591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_error -r 1 [ 1209.131954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_error -r 1 [ 1209.161362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1209.178336] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1209.413488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.438405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1209.466417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.506166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.552288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1209.607501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.643586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.679710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.723797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.741493] LNet: 62892:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.754588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.782604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.838900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.879685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.935869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.982568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1210.013887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1210.573136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_timeout -r 1 [ 1210.622563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_timeout -r 1 [ 1210.655249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1210.672133] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1210.899864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.932994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1210.971738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.013824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.052376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1211.078484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.104285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.132148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.167037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.185188] LNet: 63003:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.210742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.251887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.285812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.318880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.356502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.390038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.416196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1211.656790] LNet: 63044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1211.663964] LNetError: 63044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1211.682861] LNet: Removed LNI 192.168.204.12@tcp [ 1212.063944] Key type .llcrypt unregistered [ 1212.067968] Key type ._llcrypt unregistered [ 1227.583652] Key type ._llcrypt registered [ 1227.587501] Key type .llcrypt registered [ 1229.389835] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 07:53:08 (1788522788) [ 1230.519316] Key type .llcrypt unregistered [ 1230.522818] Key type ._llcrypt unregistered [ 1237.576855] Key type ._llcrypt registered [ 1237.579128] Key type .llcrypt registered [ 1237.737587] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1237.763757] alg: No test for adler32 (adler32-zlib) [ 1238.681687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1238.720692] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1245.463783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1253.652858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1253.681702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1253.709678] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1255.190860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_dropped -r 1 [ 1255.264162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e remote_dropped -r 1 [ 1255.322487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_dropped -r 1 [ 1255.369836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e remote_dropped -r 1 [ 1255.404824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e remote_dropped -r 1 [ 1255.456211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e remote_dropped -r 1 [ 1255.509869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e remote_dropped -r 1 [ 1255.571672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e remote_dropped -r 1 [ 1255.606482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1255.639891] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1255.656090] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.112@tcp [ 1255.923673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1255.950365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1256.002858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1256.034577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1256.065869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1256.093669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1256.146975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1256.199373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1256.290589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1256.334243] LNet: 64511:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1256.363536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1256.405389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1256.446701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.480032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.513530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.553362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.594825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1256.644568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1257.269755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_error -r 1 [ 1257.338082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e remote_error -r 1 [ 1257.386159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_error -r 1 [ 1257.451648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e remote_error -r 1 [ 1257.499410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e remote_error -r 1 [ 1257.561539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e remote_error -r 1 [ 1257.604735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e remote_error -r 1 [ 1257.649373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e remote_error -r 1 [ 1257.687268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1257.707245] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1257.713482] LNet: Skipped 2 previous similar messages [ 1257.926617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1257.952357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1257.983952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1258.024850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1258.062251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1258.105095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1258.159515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1258.214578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1258.260712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1258.288758] LNet: 64636:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1258.317227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1258.367320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1258.429557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1258.461065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1258.491912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1258.532196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1258.562596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1258.588987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1259.052574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_timeout -r 1 [ 1259.092614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e remote_timeout -r 1 [ 1259.124876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_timeout -r 1 [ 1259.171392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e remote_timeout -r 1 [ 1259.198368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e remote_timeout -r 1 [ 1259.225399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e remote_timeout -r 1 [ 1259.264493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e remote_timeout -r 1 [ 1259.294335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e remote_timeout -r 1 [ 1259.325993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.112@tcp [ 1259.356427] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1259.590873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1259.614725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1259.644567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1259.672161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1259.697932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1259.740669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.773971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.818495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.859709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.880533] LNet: 64761:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.903915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.931112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.958510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.991884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1260.035402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.068481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1260.099744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.126471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.983450] LNet: 64839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.992968] LNetError: 64839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.010713] LNet: Removed LNI 192.168.204.12@tcp [ 1265.021435] LNet: Removed LNI 192.168.204.12@tcp1 [ 1265.368744] Key type .llcrypt unregistered [ 1265.371866] Key type ._llcrypt unregistered [ 1282.001975] Key type ._llcrypt registered [ 1282.006771] Key type .llcrypt registered [ 1283.728533] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 07:54:02 (1788522842) [ 1285.224114] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1287.491297] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 07:54:06 (1788522846) [ 1288.344581] Key type .llcrypt unregistered [ 1288.354205] Key type ._llcrypt unregistered [ 1294.817897] Key type ._llcrypt registered [ 1294.820215] Key type .llcrypt registered [ 1295.035567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1295.060584] alg: No test for adler32 (adler32-zlib) [ 1296.028325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1296.083516] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1303.876718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1304.570575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e network_timeout -r 1 [ 1304.595305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e network_timeout -r 1 [ 1304.621906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1304.636655] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1304.816468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.847480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1304.891446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.933647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.981466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1305.047179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.119260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.173522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.216746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.242297] LNet: 66379:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.270266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.311386] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1305.313816] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1305.347620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.376962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.420688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.477290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.519623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.788203] LNet: 66418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1305.791965] LNetError: 66418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.853723] LNet: Removed LNI 192.168.204.12@tcp [ 1307.260373] Key type .llcrypt unregistered [ 1307.263352] Key type ._llcrypt unregistered [ 1324.904620] Key type ._llcrypt registered [ 1324.906636] Key type .llcrypt registered [ 1325.171253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1325.193517] alg: No test for adler32 (adler32-zlib) [ 1326.128793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1326.177212] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1333.547401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1342.633144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1342.681966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1342.717660] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1344.079635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e network_timeout -r 1 [ 1344.126688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e network_timeout -r 1 [ 1344.186135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e network_timeout -r 1 [ 1344.235869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e network_timeout -r 1 [ 1344.288350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e network_timeout -r 1 [ 1344.334096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e network_timeout -r 1 [ 1344.371796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e network_timeout -r 1 [ 1344.404992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e network_timeout -r 1 [ 1344.434957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1344.461029] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1344.607910] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1344.617859] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.112@tcp [ 1344.689612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1344.723350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1344.763286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1344.802885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1344.838207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1344.870793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1344.917957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1344.966735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1344.999416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1345.020306] LNet: 67295:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1345.041992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1345.072202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1345.100971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1345.132681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1345.159263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1345.190263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.361331] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1349.611759] LNet: 67369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.633675] LNetError: 67369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.639334] LNet: Removed LNI 192.168.204.12@tcp [ 1349.646650] LNet: Removed LNI 192.168.204.12@tcp1 [ 1350.042563] Key type .llcrypt unregistered [ 1350.044600] Key type ._llcrypt unregistered [ 1365.947639] Key type ._llcrypt registered [ 1365.952460] Key type .llcrypt registered [ 1367.322868] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 07:55:26 (1788522926) [ 1367.946439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1367.963435] alg: No test for adler32 (adler32-zlib) [ 1368.928636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.985411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1369.017907] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1369.698416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1369.716385] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1370.791874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1370.868586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1370.925303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1371.010832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1371.106268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1371.180813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1371.274230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.12@tcp [ 1371.311255] LNetError: 68466:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.12@tcp to recovery [ 1372.064836] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1372.068154] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1374.112173] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1374.931714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1378.207424] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1378.505396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1378.561502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1378.605728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1378.620618] LNet: 68573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.623646] LNetError: 68573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.626837] LNet: Removed LNI 192.168.204.12@tcp [ 1378.651664] LNet: Removed LNI 192.168.204.12@tcp1 [ 1378.666663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.703607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1378.719336] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1379.550328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1379.563162] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1380.632325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1380.722439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1380.861618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1380.949520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1380.993445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1381.081566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1381.147865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1381.210264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.12@tcp [ 1381.243690] LNetError: 68846:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.12@tcp to recovery [ 1381.791576] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1385.067236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1387.935450] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1387.945426] LNet: Skipped 1 previous similar message [ 1392.140469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1392.171179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1392.206538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1398.738958] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 07:55:58 (1788522958) [ 1399.124891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1399.165524] LNet: 69554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.175883] LNetError: 69554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.185689] LNetError: 69554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1399.201668] LNet: Removed LNI 192.168.204.12@tcp [ 1399.219573] LNet: Removed LNI 192.168.204.12@tcp1 [ 1399.244883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1399.318494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1399.346480] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1400.130902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1400.169331] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1401.355494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1401.438953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1401.488797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1401.545246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1401.610719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1401.666186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1401.695723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.12@tcp --health 0 [ 1401.816673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1402.335606] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.12@tcp [ 1402.335945] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1402.365947] LNet: Skipped 1 previous similar message [ 1417.536984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1417.762595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1417.805608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1417.882536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.12@tcp --health 500 [ 1418.013462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1423.875560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1423.944401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1423.955840] LNet: 70038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.964543] LNetError: 70038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.970274] LNetError: 70038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1423.976797] LNet: Removed LNI 192.168.204.12@tcp [ 1424.001340] LNet: Removed LNI 192.168.204.12@tcp1 [ 1424.027734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.134414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1424.163760] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1424.909938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1424.936472] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1426.378751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1426.437631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1426.538714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1426.607667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1426.687380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1426.764072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1426.808779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1426.858716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.12@tcp --health 0 [ 1427.167978] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1427.187650] LNet: Skipped 3 previous similar messages [ 1428.247967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1437.840615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1437.905915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1437.960366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1445.936827] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:56:44 (1788523004) [ 1446.440712] LNet: 71074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.454143] LNetError: 71074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1446.466255] LNetError: 71074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1446.472906] LNet: Removed LNI 192.168.204.12@tcp [ 1446.488277] LNet: Removed LNI 192.168.204.12@tcp1 [ 1447.366851] Key type .llcrypt unregistered [ 1447.371441] Key type ._llcrypt unregistered [ 1454.057175] Key type ._llcrypt registered [ 1454.060632] Key type .llcrypt registered [ 1454.326519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1454.354903] alg: No test for adler32 (adler32-zlib) [ 1455.375740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1455.435940] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1466.023584] Lustre: DEBUG MARKER: Initial discovery [ 1466.269371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.112@tcp [ 1469.247546] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1469.450734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1469.494288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.112@tcp [ 1469.535813] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1469.587891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1469.658085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.112@tcp -r 1 -m GET -l 3 [ 1469.735613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.112@tcp -r 1 -m GET -e local_error [ 1469.808761] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1469.831509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.112@tcp -r 1 -m PUT -l 6 [ 1471.398659] Lustre: DEBUG MARKER: Force oleg412-client.virtnet to discover 192.168.204.112@tcp (in background) [ 1471.473233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.112@tcp [ 1472.810987] Lustre: DEBUG MARKER: Force oleg412-server to discover 192.168.204.12@tcp [ 1474.528695] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1475.263622] Lustre: DEBUG MARKER: Wait for 71830 [ 1477.106210] Lustre: DEBUG MARKER: Finished wait on 71830 [ 1477.196092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1477.251837] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1477.292428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1477.716355] LNet: 72284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.733828] LNetError: 72284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.758567] LNet: Removed LNI 192.168.204.12@tcp [ 1479.276799] Key type .llcrypt unregistered [ 1479.279980] Key type ._llcrypt unregistered [ 1498.846918] Key type ._llcrypt registered [ 1498.850449] Key type .llcrypt registered [ 1500.455787] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:57:39 (1788523059) [ 1501.053599] Key type .llcrypt unregistered [ 1501.056084] Key type ._llcrypt unregistered [ 1507.546547] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1507.794949] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1507.799673] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1507.802290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1510.262387] Key type ._llcrypt registered [ 1510.265150] Key type .llcrypt registered [ 1510.412208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1510.423188] alg: No test for adler32 (adler32-zlib) [ 1511.359931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.417364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1511.440817] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1512.443836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1512.497342] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1521.238506] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:58:00 (1788523080) [ 1521.348478] LNet: 74349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.356764] LNetError: 74349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1521.362257] LNet: Removed LNI 10.1.2.3@tcp [ 1521.378285] LNet: Removed LNI 192.168.204.12@tcp [ 1521.953632] Key type .llcrypt unregistered [ 1521.958522] Key type ._llcrypt unregistered [ 1528.565026] Key type ._llcrypt registered [ 1528.571153] Key type .llcrypt registered [ 1528.938418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1528.984760] alg: No test for adler32 (adler32-zlib) [ 1530.028942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.102832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.136563] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1530.872610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1530.897741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1531.951108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.12@tcp [ 1533.023375] LNetError: Unexpected error -22 connecting to 192.168.204.12@tcp at host 192.168.204.12:988 [ 1538.166780] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:58:17 (1788523097) [ 1538.320763] LNet: 75480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.350914] LNetError: 75480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1538.357430] LNet: Removed LNI 10.1.2.3@tcp [ 1538.374774] LNet: Removed LNI 192.168.204.12@tcp [ 1539.277336] Key type .llcrypt unregistered [ 1539.284709] Key type ._llcrypt unregistered [ 1545.301368] Key type ._llcrypt registered [ 1545.305397] Key type .llcrypt registered [ 1545.633161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1545.658763] alg: No test for adler32 (adler32-zlib) [ 1546.624561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.679690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1546.702959] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1547.650169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1547.677927] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 1548.822057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.12@tcp1 --nid 192.168.204.12@tcp2 [ 1554.296250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp1 192.168.204.12@tcp2 [ 1554.325695] LNetError: 76318:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.12@tcp1 to 192.168.204.12@tcp2 - Route table may be misconfigured [ 1554.359082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp2 192.168.204.12@tcp1 [ 1561.257994] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:58:40 (1788523120) [ 1561.485579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.543017] LNet: 76891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.562847] LNetError: 76891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.578183] LNet: Removed LNI 192.168.204.12@tcp1 [ 1562.600878] LNet: Removed LNI 192.168.204.12@tcp2 [ 1562.646655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.728347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.760907] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1563.560870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1563.593657] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1564.538349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 1564.592987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.12@tcp [ 1564.623979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp1 192.168.204.12@tcp1 [ 1564.693825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.12@tcp -d 192.168.204.12@tcp -e network_timeout [ 1564.743661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -e network_timeout [ 1564.795792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -e network_timeout [ 1564.838967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -e network_timeout [ 1564.885051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1564.913392] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1564.947907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1565.095819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1565.240961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1565.792255] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1572.405513] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:58:51 (1788523131) [ 1572.521987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.555702] LNet: 77745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.581696] LNetError: 77745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.599644] LNetError: 77745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1572.613764] LNet: Removed LNI 192.168.204.12@tcp [ 1573.680402] LNet: Removed LNI 192.168.204.12@tcp1 [ 1573.712866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.787781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1574.192958] LNet: 77790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.722687] Key type .llcrypt unregistered [ 1574.727861] Key type ._llcrypt unregistered [ 1586.824912] Key type ._llcrypt registered [ 1586.829343] Key type .llcrypt registered [ 1588.211853] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:59:07 (1788523147) [ 1588.615209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1588.636480] alg: No test for adler32 (adler32-zlib) [ 1589.614395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.047292] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1592.608711] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:59:12 (1788523152) [ 1592.720907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1592.746527] LNet: 78880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1592.777663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.867613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1592.895574] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1593.651238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1593.683750] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1594.486905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1594.512825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp1 [ 1594.539634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp1 [ 1600.803770] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:59:20 (1788523160) [ 1602.795339] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg412-server [ 1604.671452] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:59:24 (1788523164) [ 1606.422487] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg412-server [ 1608.101514] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:59:27 (1788523167) [ 1609.939720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg412-server [ 1611.632482] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:59:31 (1788523171) [ 1613.406060] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg412-server [ 1615.245423] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:59:34 (1788523174) [ 1617.019459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg412-server [ 1618.821781] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:59:38 (1788523178) [ 1621.000882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg412-server [ 1623.282315] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:59:42 (1788523182) [ 1626.004723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg412-server [ 1627.687933] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:59:47 (1788523187) [ 1629.930923] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg412-server [ 1631.721098] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:59:51 (1788523191) [ 1633.893315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg412-server [ 1635.761718] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:59:55 (1788523195) [ 1635.982321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.009964] LNet: 82762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.036780] LNetError: 82762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.046305] LNet: Removed LNI 192.168.204.12@tcp [ 1637.115249] LNet: Removed LNI 192.168.204.12@tcp1 [ 1637.171795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.252671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1637.269159] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1637.984511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1638.088643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1638.363227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.426934] LNet: 82910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.438180] LNetError: 82910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.445990] LNetError: 82910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1638.478065] LNet: Removed LNI 192.168.204.12@tcp [ 1638.502764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.567308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1638.592636] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1639.554203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1639.860920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.918672] LNetError: 83055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.942973] LNet: Removed LNI 192.168.204.12@tcp [ 1639.976396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1640.072445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1640.156549] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1641.177866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1641.290725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1641.583254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1641.632480] LNet: 83202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.639617] LNet: 83202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1642.669106] LNet: Removed LNI 192.168.204.12@tcp [ 1642.720911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.819392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1642.844978] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1643.669137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1643.968537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1643.993173] LNetError: 83348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1643.998373] LNetError: 83348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1644.012931] LNet: Removed LNI 192.168.204.12@tcp [ 1644.050975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.138873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.194288] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1645.111861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1645.229481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1645.516188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.573371] LNet: Removed LNI 192.168.204.12@tcp [ 1646.638555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.753220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1646.795501] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1647.841114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1648.092793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.143421] LNet: 83640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.146445] LNet: 83640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1648.162652] LNetError: 83640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1648.175192] LNetError: 83640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1648.190298] LNet: Removed LNI 192.168.204.12@tcp [ 1648.219537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.336990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1648.387799] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1649.418555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1649.555318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1649.853771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.919283] LNet: Removed LNI 192.168.204.12@tcp [ 1650.956867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.051558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1651.091683] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1651.950628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1652.237819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.291042] LNet: Removed LNI 192.168.204.12@tcp [ 1653.317271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.399350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1653.428605] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1654.070766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1654.145620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1655.591465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.683356] LNet: Removed LNI 192.168.204.12@tcp [ 1656.746491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.856046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1656.912751] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1657.936707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1658.332775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.364887] LNet: 84235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.371029] LNet: 84235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1658.384158] LNetError: 84235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.393106] LNetError: 84235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1658.412656] LNet: Removed LNI 192.168.204.12@tcp [ 1658.429685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.491474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.521313] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1659.172427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1659.287325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1659.661062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.710975] LNet: Removed LNI 192.168.204.12@tcp [ 1660.740201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.829570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1660.883858] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1661.727518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1662.146695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.210804] LNet: Removed LNI 192.168.204.12@tcp [ 1663.278684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.409174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1663.461391] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1664.830935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1664.908043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.941470] LNet: Removed LNI 192.168.204.12@tcp [ 1664.961444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.016947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.042124] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1666.001274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1666.065529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.090434] LNet: Removed LNI 192.168.204.12@tcp [ 1666.116710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.169618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1666.254303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.314622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.360516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1666.424406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.457727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.533576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1666.670245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.703516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.765668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1666.811304] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1667.909892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.939586] LNet: Removed LNI 192.168.204.12@tcp [ 1667.959160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.034261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1676.563823] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 08:00:35 (1788523235) [ 1676.772535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.856889] LNet: 85545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.866319] LNet: 85545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1676.919422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.067518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1677.125675] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/180] [ 1678.950589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.989797] LNetError: 85688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.998389] LNetError: 85688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1679.014683] LNet: Removed LNI 192.168.204.12@tcp231 [ 1679.040962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.118128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1679.150282] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/180] [ 1679.155424] LNet: 85696:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1679.642438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1679.682504] LNet: Removed LNI 192.168.204.12@tcp231 [ 1679.718781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1679.749100] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/180] [ 1681.396885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.429472] LNet: Removed LNI 192.168.204.12@tcp231 [ 1681.448896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.498356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1681.515745] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/0] [ 1681.519415] LNet: 85849:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1681.730621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1681.766096] LNet: Removed LNI 192.168.204.12@tcp231 [ 1681.781820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1681.812606] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/0] [ 1682.789660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.830018] LNet: Removed LNI 192.168.204.12@tcp231 [ 1682.859674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.954951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1682.980793] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/60] [ 1682.986646] LNet: 86000:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1683.473772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1683.545751] LNet: Removed LNI 192.168.204.12@tcp231 [ 1683.562111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1683.592084] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/60] [ 1684.771211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.814508] LNet: Removed LNI 192.168.204.12@tcp231 [ 1684.865788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.930535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1684.954674] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/180] [ 1685.287694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1685.308213] LNet: Removed LNI 192.168.204.12@tcp231 [ 1685.322753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1685.345028] LNet: Added LNI 192.168.204.12@tcp231 [8/256/0/180] [ 1694.827347] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 08:00:54 (1788523254) [ 1695.124954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.150670] LNet: Removed LNI 192.168.204.12@tcp231 [ 1695.180266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.249560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1695.274825] LNet: Added LNI 192.168.204.12@tcp232 [8/256/0/180] [ 1696.118596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1696.189350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1696.231689] LNet: Removed LNI 192.168.204.12@tcp232 [ 1696.280781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1696.335270] LNet: Added LNI 192.168.204.12@tcp232 [8/256/0/180] [ 1697.178836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1697.254511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1697.282913] LNet: Removed LNI 192.168.204.12@tcp232 [ 1697.305604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1697.355207] LNet: Added LNI 192.168.204.12@tcp232 [8/256/0/180] [ 1698.031882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1698.091368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1698.125221] LNet: Removed LNI 192.168.204.12@tcp232 [ 1698.146454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1698.175335] LNet: Added LNI 192.168.204.12@tcp232 [8/256/0/180] [ 1699.072501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1699.151430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1699.196234] LNet: Removed LNI 192.168.204.12@tcp232 [ 1699.236850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1707.137668] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 08:01:06 (1788523266) [ 1708.934937] Key type .llcrypt unregistered [ 1708.940320] Key type ._llcrypt unregistered [ 1716.221677] Key type ._llcrypt registered [ 1716.227538] Key type .llcrypt registered [ 1716.408660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1716.423880] alg: No test for adler32 (adler32-zlib) [ 1717.462399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1717.513309] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1725.213655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1735.993582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.081667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1736.118086] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1737.797403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_interrupt -r 2 [ 1737.859528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_interrupt -r 2 [ 1737.909642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_interrupt -r 2 [ 1737.955097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_interrupt -r 2 [ 1738.000961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_interrupt -r 2 [ 1738.068421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_interrupt -r 2 [ 1738.134928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_interrupt -r 2 [ 1738.206615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_interrupt -r 2 [ 1738.256483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1738.280507] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1738.289165] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1738.600056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1738.627205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1738.652728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1738.681050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1738.711907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1738.754217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1738.797558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1738.847798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1738.893852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1738.918988] LNet: 88669:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1738.943409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1738.981155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1739.019703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1739.064834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1739.112791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1739.154505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1739.694294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_dropped -r 2 [ 1739.723219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_dropped -r 2 [ 1739.753639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_dropped -r 2 [ 1739.782383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_dropped -r 2 [ 1739.808463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_dropped -r 2 [ 1739.840391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_dropped -r 2 [ 1739.884731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_dropped -r 2 [ 1739.928430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_dropped -r 2 [ 1739.960561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1739.979499] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1740.280740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1740.316642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1740.367775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1740.411821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1740.469914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1740.533688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1740.583593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1740.629661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1740.654984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1740.670286] LNet: 88791:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1740.684839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1740.709371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1740.733567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1740.772916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1740.813578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1740.882584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1741.461918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_aborted -r 2 [ 1741.513791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_aborted -r 2 [ 1741.557951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_aborted -r 2 [ 1741.613349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_aborted -r 2 [ 1741.657433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_aborted -r 2 [ 1741.697501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_aborted -r 2 [ 1741.739585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_aborted -r 2 [ 1741.784663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_aborted -r 2 [ 1741.819665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1741.835982] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1742.053446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1742.086984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1742.122639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1742.157489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1742.185989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1742.219384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1742.247070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1742.281918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1742.305425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1742.321394] LNet: 88913:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1742.336423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1742.364578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1742.401936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1742.433747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1742.469542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1742.505701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1743.034797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_no_route -r 2 [ 1743.072744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_no_route -r 2 [ 1743.104413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_no_route -r 2 [ 1743.134664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_no_route -r 2 [ 1743.165985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_no_route -r 2 [ 1743.198378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_no_route -r 2 [ 1743.238230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_no_route -r 2 [ 1743.277995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_no_route -r 2 [ 1743.321859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1743.596337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1743.618154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1743.640539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1743.664956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1743.688844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1743.710351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1743.751043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1743.808656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1743.844342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1743.868989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1743.906540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1743.954109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1744.012716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1744.062323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1744.111081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1744.612731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e local_timeout -r 2 [ 1744.643377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e local_timeout -r 2 [ 1744.673729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e local_timeout -r 2 [ 1744.706766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e local_timeout -r 2 [ 1744.739391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e local_timeout -r 2 [ 1744.768929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e local_timeout -r 2 [ 1744.808132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e local_timeout -r 2 [ 1744.838430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e local_timeout -r 2 [ 1744.873977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1744.894500] LNet: There was an unexpected network error while writing to 192.168.204.112: rc = -22 [ 1744.904041] LNet: Skipped 1 previous similar message [ 1745.211543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1745.236833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1745.265606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1745.289356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1745.316605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1745.344457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1745.369572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1745.400410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1745.440888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1745.476559] LNet: 89158:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1745.486516] LNet: 89158:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1745.516552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1745.557508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1745.582438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1745.620436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1745.648634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1745.687149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1746.131657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp -m GET -e remote_dropped -r 2 [ 1746.170876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.112@tcp1 -m GET -e remote_dropped -r 2 [ 1746.206464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp -m GET -e remote_dropped -r 2 [ 1746.245874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp -d 192.168.204.12@tcp1 -m GET -e remote_dropped -r 2 [ 1746.281471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp -m GET -e remote_dropped -r 2 [ 1746.316117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.112@tcp1 -m GET -e remote_dropped -r 2 [ 1746.367140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp -m GET -e remote_dropped -r 2 [ 1746.401783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.12@tcp1 -d 192.168.204.12@tcp1 -m GET -e remote_dropped -r 2 [ 1746.437389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.12@tcp 192.168.204.112@tcp [ 1746.454441] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.112@tcp [ 1746.668871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1746.698762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1746.736477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1746.766439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1746.793146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1746.829819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1746.878659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1746.918079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1746.964053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1747.017217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1747.061470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1747.094226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1747.137365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1747.167682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1747.203880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1752.666399] LNet: 89354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.680606] LNetError: 89354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.687817] LNet: Removed LNI 192.168.204.12@tcp [ 1752.699833] LNet: Removed LNI 192.168.204.12@tcp1 [ 1753.019728] Key type .llcrypt unregistered [ 1753.026650] Key type ._llcrypt unregistered [ 1767.453196] Key type ._llcrypt registered [ 1767.455770] Key type .llcrypt registered [ 1768.725893] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 08:02:08 (1788523328) [ 1769.071763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1769.084926] alg: No test for adler32 (adler32-zlib) [ 1769.988777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.038194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1770.057807] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1770.870568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1770.922489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.12@tcp [ 1770.937855] LNetError: 90332:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.12@tcp to recovery [ 1771.030950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.12@tcp [ 1771.043204] LNet: 1 local NIs in recovery (showing 1): 192.168.204.12@tcp [ 1771.145262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1776.537859] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 08:02:16 (1788523336) [ 1776.621825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.638536] LNet: 90915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.651166] LNetError: 90915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.715366] LNet: Removed LNI 192.168.204.12@tcp [ 1777.746859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1777.816996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1777.868484] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1778.518850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1778.543838] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 1779.617874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.12@tcp [ 1779.643761] LNetError: 91188:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.204.12@tcp to recovery [ 1779.809367] LNet: There was an unexpected network error while writing to 192.168.204.12: rc = -22 [ 1780.999590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.12@tcp [ 1789.669621] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 08:02:28 (1788523348) [ 1791.994197] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg412-server [ 1795.215691] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 08:02:33 (1788523353) [ 1798.281541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg412-server [ 1801.029839] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 08:02:39 (1788523359) [ 1804.293920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg412-server [ 1806.950964] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 08:02:45 (1788523365) [ 1807.221090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.280112] LNet: 92809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.335706] LNetError: 92809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.352453] LNet: Removed LNI 192.168.204.12@tcp [ 1807.369914] LNet: Removed LNI 192.168.204.12@tcp1 [ 1807.418190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.498859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1807.532259] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1808.860880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1808.975078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1809.058263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1809.244360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1809.314932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1809.447171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1817.203892] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 08:02:56 (1788523376) [ 1817.655304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.690964] LNet: 93541:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.701870] LNetError: 93541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.717573] LNetError: 93541:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1817.742320] LNet: Removed LNI 192.168.204.12@tcp [ 1817.778463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.876107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1817.895675] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1826.990616] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 08:03:06 (1788523386) [ 1827.366033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.411991] LNet: 94246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.420178] LNetError: 94246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1827.434976] LNet: Removed LNI 192.168.204.12@tcp [ 1827.450792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.519371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.597149] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1836.716456] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 08:03:16 (1788523396) [ 1838.654640] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1840.490937] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 08:03:20 (1788523400) [ 1841.252841] LNet: 95281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.261079] LNetError: 95281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.279299] LNet: Removed LNI 192.168.204.12@tcp [ 1842.088982] Key type .llcrypt unregistered [ 1842.092616] Key type ._llcrypt unregistered [ 1849.205311] Key type ._llcrypt registered [ 1849.209497] Key type .llcrypt registered [ 1849.425437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1849.442312] alg: No test for adler32 (adler32-zlib) [ 1850.360203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1850.385673] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1858.804865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 1864.068851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.112@tcp [ 1864.404366] LNet: 95696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.410683] LNetError: 95696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.425794] LNet: Removed LNI 192.168.204.12@tcp [ 1864.871598] Key type .llcrypt unregistered [ 1864.878519] Key type ._llcrypt unregistered [ 1876.761355] Key type ._llcrypt registered [ 1876.763777] Key type .llcrypt registered [ 1877.940896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1879.743764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1881.286437] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 08:04:00 (1788523440) [ 1882.936938] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1884.993881] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 08:04:04 (1788523444) [ 1887.059353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg412-server [ 1888.709225] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 08:04:08 (1788523448) [ 1890.950321] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg412-server [ 1893.085351] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 08:04:12 (1788523452) [ 1894.765303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1894.815508] alg: No test for adler32 (adler32-zlib) [ 1896.162388] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 1896.417269] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1898.103176] Key type lgssc registered [ 1899.467219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1918.721973] Key type lgssc unregistered [ 1919.012445] LNet: 98866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.033685] LNetError: 98866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.071420] LNet: Removed LNI 192.168.204.12@tcp [ 1919.798221] Key type .llcrypt unregistered [ 1919.802065] Key type ._llcrypt unregistered [ 1935.091339] Key type ._llcrypt registered [ 1935.096120] Key type .llcrypt registered [ 1936.213056] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 08:04:55 (1788523495) [ 1936.695353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1936.708954] alg: No test for adler32 (adler32-zlib) [ 1937.625079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.674242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1937.723305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1937.788549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1937.820895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1943.508283] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 08:05:03 (1788523503) [ 1943.617076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.651669] LNet: 100272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.686777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.766566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.801316] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1944.793478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.808931] LNet: 100423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.814554] LNetError: 100423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.832038] LNet: Removed LNI 192.168.204.12@tcp [ 1944.852705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.902716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1944.929764] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1945.415682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1945.438454] LNetError: 100451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.453784] LNet: Removed LNI 192.168.204.12@tcp [ 1945.476192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.522840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1945.545840] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1950.750576] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 08:05:10 (1788523510) [ 1951.117456] LNet: 101041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.122477] LNet: 101041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1951.145474] LNetError: 101041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.163515] LNet: Removed LNI 192.168.204.12@tcp [ 1951.778728] Key type .llcrypt unregistered [ 1951.784179] Key type ._llcrypt unregistered [ 1958.323831] Key type ._llcrypt registered [ 1958.325599] Key type .llcrypt registered [ 1958.610106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1958.641635] alg: No test for adler32 (adler32-zlib) [ 1959.525751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.584098] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 1959.586326] LNet: *** cfs_fail_loc=e003, val=100*** [ 1960.150453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 1960.167871] LNet: *** cfs_fail_loc=e003, val=100*** [ 1960.170943] LNet: Skipped 2 previous similar messages [ 1960.176357] Ping reply received with mapping: [ 1960.179351] found metadata reply NI[192.168.204.12@tcp] [ 1960.182358] 0c cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1960.185943] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1960.189369] 5a 5a 5a 5a 5a 5a 5a 5a [ 1960.350571] LNet: 101435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.359262] LNetError: 101435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.375771] LNet: Removed LNI 192.168.204.12@tcp [ 1961.019666] Key type .llcrypt unregistered [ 1961.023914] Key type ._llcrypt unregistered [ 1972.024507] Key type ._llcrypt registered [ 1972.026352] Key type .llcrypt registered [ 1973.184980] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 08:05:32 (1788523532) [ 1973.764412] Key type .llcrypt unregistered [ 1973.769504] Key type ._llcrypt unregistered [ 1980.788949] Key type ._llcrypt registered [ 1980.797348] Key type .llcrypt registered [ 1981.059451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1981.074845] alg: No test for adler32 (adler32-zlib) [ 1982.170261] LNetError: 102471:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1982.511485] Key type .llcrypt unregistered [ 1982.515542] Key type ._llcrypt unregistered [ 1987.481957] Key type ._llcrypt registered [ 1987.485298] Key type .llcrypt registered [ 1988.966342] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 08:05:48 (1788523548) [ 1989.509100] Key type .llcrypt unregistered [ 1989.510821] Key type ._llcrypt unregistered [ 1995.856252] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1995.865035] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1998.517584] Key type ._llcrypt registered [ 1998.522698] Key type .llcrypt registered [ 1998.700505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1998.726502] alg: No test for adler32 (adler32-zlib) [ 1999.642205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1999.711420] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2009.355979] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 08:06:08 (1788523568) [ 2009.500128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2009.528261] LNet: 104239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.536949] LNetError: 104239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2009.553269] LNet: Removed LNI 10.1.2.3@tcp [ 2009.574080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.488744] Key type .llcrypt unregistered [ 2010.490962] Key type ._llcrypt unregistered [ 2017.980939] Key type ._llcrypt registered [ 2017.983856] Key type .llcrypt registered [ 2018.251928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2018.275950] alg: No test for adler32 (adler32-zlib) [ 2019.194755] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2019.288931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2019.445804] LNet: 104532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.451246] LNetError: 104532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2019.475723] LNet: Removed LNI 192.168.204.12@tcp [ 2020.294701] Key type .llcrypt unregistered [ 2020.296472] Key type ._llcrypt unregistered [ 2033.948741] Key type ._llcrypt registered [ 2033.952233] Key type .llcrypt registered [ 2036.074975] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 08:06:34 (1788523594) [ 2037.049544] Key type .llcrypt unregistered [ 2037.052935] Key type ._llcrypt unregistered [ 2048.531610] Key type ._llcrypt registered [ 2048.533819] Key type .llcrypt registered [ 2048.667682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2048.685968] alg: No test for adler32 (adler32-zlib) [ 2057.438683] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 08:06:56 (1788523616) [ 2057.729325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2057.774432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2057.831492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2057.852193] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2058.969804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.008579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2059.040809] LNet: Removed LNI 192.168.204.12@tcp [ 2059.437829] LNet: 106272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.964562] Key type .llcrypt unregistered [ 2059.966956] Key type ._llcrypt unregistered [ 2073.251477] Key type ._llcrypt registered [ 2073.254628] Key type .llcrypt registered [ 2074.789325] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 08:07:13 (1788523633) [ 2075.266543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.292966] alg: No test for adler32 (adler32-zlib) [ 2076.199301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.245559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2076.274638] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2076.983039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2077.044379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.12@tcp [ 2077.062091] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2077.065551] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2077.069064] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2077, 0, 0 [ 2077.075937] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2077.089362] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2077.096413] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2077.109442] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2077, 0, 0 [ 2077.119654] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2077.123690] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2077.130685] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2077.139190] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2077, 0, 2127 [ 2077.152716] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2077.156782] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2077.161606] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2077.166357] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2077, 0, 2127 [ 2077.170706] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2083.974454] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 08:07:23 (1788523643) [ 2084.586477] LNet: 107809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2084.601574] LNetError: 107809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.621978] LNet: Removed LNI 192.168.204.12@tcp [ 2085.225686] Key type .llcrypt unregistered [ 2085.231500] Key type ._llcrypt unregistered [ 2091.932436] Key type ._llcrypt registered [ 2091.934533] Key type .llcrypt registered [ 2092.185352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2092.204417] alg: No test for adler32 (adler32-zlib) [ 2093.165712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2093.204398] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2100.280322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 2109.917913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2109.961504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2109.979839] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 2110.957070] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2111.108965] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2111.115987] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2111.121740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2113.527164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2113.565381] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2124.817703] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 08:08:03 (1788523683) [ 2124.889427] LNet: 109308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.899821] LNetError: 109308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.912532] LNet: Removed LNI 192.168.204.12@tcp [ 2125.986985] LNet: Removed LNI 192.168.204.12@tcp1 [ 2125.988676] LNetError: 109308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2125.993115] LNetError: 109308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2126.014411] LNet: Removed LNI 10.1.2.3@tcp99 [ 2126.550313] Key type .llcrypt unregistered [ 2126.552334] Key type ._llcrypt unregistered [ 2132.618946] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2132.633362] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2134.162077] Key type ._llcrypt registered [ 2134.164960] Key type .llcrypt registered [ 2134.310371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2134.325959] alg: No test for adler32 (adler32-zlib) [ 2135.340804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.434516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2135.467842] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2136.285359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2136.301397] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2138.446515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2138.511361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2138.579400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2138.617497] LNetError: 110050:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2138.681854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2138.753407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2139.286554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2139.359691] LNet: 110070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.365915] LNetError: 110070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2139.371911] LNet: Removed LNI 10.1.2.3@tcp [ 2140.390748] LNet: Removed LNI 192.168.204.12@tcp [ 2141.084639] Key type .llcrypt unregistered [ 2141.087648] Key type ._llcrypt unregistered [ 2152.989839] Key type ._llcrypt registered [ 2152.992655] Key type .llcrypt registered [ 2154.580517] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 08:08:33 (1788523713) [ 2155.017808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2155.047407] alg: No test for adler32 (adler32-zlib) [ 2155.927186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.965818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.986497] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2156.568754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2156.630494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg412-client@tcp [ 2162.520814] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 08:08:42 (1788523722) [ 2162.740664] LNet: 111600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2162.750154] LNetError: 111600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2162.767813] LNet: Removed LNI 192.168.204.12@tcp [ 2163.481555] Key type .llcrypt unregistered [ 2163.484408] Key type ._llcrypt unregistered [ 2170.732335] Key type ._llcrypt registered [ 2170.737913] Key type .llcrypt registered [ 2170.919507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2170.937162] alg: No test for adler32 (adler32-zlib) [ 2171.852312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2171.872415] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2171.896502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2178.356203] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 08:08:57 (1788523737) [ 2179.827644] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2181.982986] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 08:09:01 (1788523741) [ 2183.697767] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2185.430322] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 08:09:04 (1788523744) [ 2185.563365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2185.609203] LNet: 113018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2185.625241] LNetError: 113018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.640907] LNet: Removed LNI 192.168.204.12@tcp [ 2185.656875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2185.727161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2185.759559] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2186.505237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2186.544757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2193.498243] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 08:09:13 (1788523753) [ 2193.634255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.668612] LNet: 113729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.682590] LNetError: 113729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.714916] LNet: Removed LNI 192.168.204.12@tcp [ 2193.747687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.802551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2193.825630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2194.341523] Key type .llcrypt unregistered [ 2194.343071] Key type ._llcrypt unregistered [ 2206.438459] Key type ._llcrypt registered [ 2206.448693] Key type .llcrypt registered [ 2207.744346] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 08:09:27 (1788523767) [ 2208.191844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2208.213880] alg: No test for adler32 (adler32-zlib) [ 2209.193994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.251700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.278884] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2209.957526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2210.029836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2217.277124] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 08:09:36 (1788523776) [ 2217.341185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.362900] LNet: 115278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2217.378716] LNetError: 115278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.405333] LNet: Removed LNI 192.168.204.12@tcp [ 2218.439045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.496581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2218.544331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2225.056812] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 08:09:44 (1788523784) [ 2225.151396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2225.169797] LNet: 115861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2225.204827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.312891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2225.775577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2225.805441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2225.900590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2226.426763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2226.460266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2226.573278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.596973] LNet: 115899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.615461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.698532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2227.086443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2227.125619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2227.183408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2227.636582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2227.703469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2233.777106] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 08:09:53 (1788523793) [ 2233.978137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.036668] LNet: 116499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.066972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2234.147103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2234.165271] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2234.880778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2234.895870] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2235.713916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2235.792674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2236.043590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2236.072550] LNetError: 116778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.081340] LNet: Removed LNI 192.168.204.12@tcp [ 2236.092856] LNet: Removed LNI 192.168.204.12@tcp2 [ 2236.111847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.172617] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2236.177176] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2236.182816] LNet: 116784:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2236.798550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2237.054106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.078699] LNetError: 116821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2237.084821] LNetError: 116821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2237.092390] LNet: Removed LNI 192.168.204.12@tcp [ 2237.109370] LNet: Removed LNI 192.168.204.12@tcp2 [ 2237.125873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.195792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2237.211769] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2237.768195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2237.798115] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2238.717211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2238.761121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2238.854503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2239.137627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2239.158783] LNet: 117106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.163708] LNet: 117106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2239.170676] LNetError: 117106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.178166] LNetError: 117106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2239.185347] LNet: Removed LNI 192.168.204.12@tcp [ 2240.239502] LNet: Removed LNI 192.168.204.12@tcp2 [ 2240.273818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.368810] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2240.376679] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2240.381511] LNet: 117112:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2240.969723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2241.189255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2241.206237] LNet: Removed LNI 192.168.204.12@tcp [ 2242.279995] LNet: Removed LNI 192.168.204.12@tcp2 [ 2242.334273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2242.420417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2242.439914] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2243.072966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2243.092368] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2244.106447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2244.192937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2244.444513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.466408] LNetError: 117429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2244.471465] LNetError: 117429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2244.477706] LNet: Removed LNI 192.168.204.12@tcp [ 2244.488251] LNet: Removed LNI 192.168.204.12@tcp2 [ 2244.506656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.580758] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2244.590307] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2244.598718] LNet: 117435:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2245.215862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2245.422804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2246.495567] LNet: Removed LNI 192.168.204.12@tcp [ 2246.511045] LNet: Removed LNI 192.168.204.12@tcp2 [ 2246.546363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.641918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2246.673772] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2247.317808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2247.353220] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2248.120042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2248.171151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2248.243442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2248.402786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.428443] LNet: 117756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.435531] LNet: 117756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2249.505250] LNet: Removed LNI 192.168.204.12@tcp [ 2250.539597] LNet: Removed LNI 192.168.204.12@tcp2 [ 2250.590170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.717853] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2250.733190] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2250.747460] LNet: 117763:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2251.359525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2251.575415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.603326] LNet: Removed LNI 192.168.204.12@tcp [ 2252.655647] LNet: Removed LNI 192.168.204.12@tcp2 [ 2252.691950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.765467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2252.802923] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2253.471739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2253.502567] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2254.337212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2254.369986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2254.414561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2254.684668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2254.698385] LNetError: 118098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2254.702254] LNetError: 118098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2254.707551] LNet: Removed LNI 192.168.204.12@tcp [ 2254.715895] LNet: Removed LNI 192.168.204.12@tcp2 [ 2254.735160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.813153] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2254.818349] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2254.824647] LNet: 118104:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2255.352419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2255.672566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.735257] LNet: Removed LNI 192.168.204.12@tcp [ 2256.750748] LNet: Removed LNI 192.168.204.12@tcp2 [ 2256.828421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.897613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.942137] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2257.575318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2257.597954] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2258.383598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2258.421182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2258.447425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2258.515850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2258.855170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.871981] LNet: Removed LNI 192.168.204.12@tcp [ 2260.968725] LNet: Removed LNI 192.168.204.12@tcp2 [ 2261.002043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.087711] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2261.095718] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2261.530432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2268.272843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2270.327643] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 08:10:29 (1788523829) [ 2270.405721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.433978] LNet: 119217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.437919] LNet: 119217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2270.442700] LNet: Removed LNI 192.168.204.12@tcp [ 2271.464601] LNet: Removed LNI 192.168.204.12@tcp2 [ 2271.513599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.044859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2272.098630] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2272.890606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2272.908818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2273.848078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.863971] LNetError: 119496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2273.871601] LNetError: 119496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2273.878537] LNet: Removed LNI 10.1.2.3@tcp [ 2273.891756] LNet: Removed LNI 192.168.204.12@tcp [ 2273.905305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.972286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2273.991693] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2273.996942] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2274.345886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2274.366653] LNet: Removed LNI 192.168.204.12@tcp [ 2274.376663] LNet: Removed LNI 10.1.2.3@tcp [ 2281.237635] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 08:10:40 (1788523840) [ 2281.495595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.538715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.602886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2281.635762] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2282.493735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2282.521675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2282.566747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2282.605923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2282.642660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2282.697801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2282.750799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2282.799893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2282.844830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2282.889431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2282.934950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2282.982793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2283.188792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2290.666735] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 08:10:50 (1788523850) [ 2291.074479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2292.200141] LNet: Removed LNI 192.168.204.12@tcp [ 2292.262375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.347337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2292.409680] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2293.756674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2293.825783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2293.942966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2294.097858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2294.234433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2294.326771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2294.418314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2294.486968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2294.561523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2294.656377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2294.715250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2294.791314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2294.957946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2295.049716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2295.119878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2295.211047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2295.322906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2295.398908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2295.456695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2295.498989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2295.545407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2295.605170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2295.713762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.12@tcp [ 2303.201484] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 08:11:02 (1788523862) [ 2303.302856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.372562] LNet: 121587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.375151] LNet: 121587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2304.424762] LNet: Removed LNI 192.168.204.12@tcp [ 2304.461832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.623928] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2304.758654] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2304.770438] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2304.795201] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2307.304345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2307.325849] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2308.244749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2308.279559] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2310.604415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2310.660372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2310.742940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.12@tcp -m PUT -e local_timeout -r 1 [ 2310.838745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2310.921865] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2310.926403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2311.261350] LNetError: 121660:0:(lib-move.c:4794:lnet_parse()) 192.168.204.12@tcp, src 192.168.204.12@tcp: Dropping PUT (error -108 looking up sender) [ 2311.979404] LNetError: 122119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2312.006918] LNetError: 122119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2312.015924] LNet: Removed LNI 10.1.2.3@tcp [ 2313.073962] LNet: Removed LNI 192.168.204.12@tcp [ 2313.639451] Key type .llcrypt unregistered [ 2313.647144] Key type ._llcrypt unregistered [ 2325.403805] Key type ._llcrypt registered [ 2325.406484] Key type .llcrypt registered [ 2326.556569] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 08:11:26 (1788523886) [ 2326.903058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2326.929151] alg: No test for adler32 (adler32-zlib) [ 2327.851144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.338739] LNet: 122970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.049883] Key type .llcrypt unregistered [ 2329.053599] Key type ._llcrypt unregistered [ 2335.295189] Key type ._llcrypt registered [ 2335.297472] Key type .llcrypt registered [ 2335.486703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2335.509404] alg: No test for adler32 (adler32-zlib) [ 2336.384185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2336.408086] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2344.261184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 2344.485707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2345.048965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.112@tcp [ 2345.314444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2345.348301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2345.385973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2345.435502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2345.478136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2345.509684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2345.542243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2345.567915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2345.598489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2345.616742] LNet: 123419:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2345.633145] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.112@tcp [ 2345.644045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2345.686088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2345.719046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2345.750538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2345.804971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2345.852988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2346.617761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2346.890480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2346.923601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2346.951284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2346.975509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2347.000033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2347.027486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2347.068936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2347.105799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2347.148825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2347.176302] LNet: 123538:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.207766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2347.253331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2347.281148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2347.307789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2347.337310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2347.385979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2348.111105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.112@tcp [ 2348.428731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2348.476622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2348.504994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2348.526707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2348.568255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2348.596846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2348.620455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2348.646941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2348.692062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2348.720939] LNet: 123658:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2348.746811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2348.800978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2348.844647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2348.886534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2348.944585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2349.004241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2349.888511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2350.170585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2350.213792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2350.246698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2350.287800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2350.319712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2350.350241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2350.380954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2350.421918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2350.461413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2350.506754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2350.554581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2350.594872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2350.633796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2350.664640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2350.700871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2351.100560] LNet: 123823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2351.106375] LNetError: 123823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2351.122488] LNet: Removed LNI 192.168.204.12@tcp [ 2351.604793] Key type .llcrypt unregistered [ 2351.608616] Key type ._llcrypt unregistered [ 2374.734918] Key type ._llcrypt registered [ 2374.740699] Key type .llcrypt registered [ 2376.652554] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 08:12:15 (1788523935) [ 2377.246299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2377.271269] alg: No test for adler32 (adler32-zlib) [ 2378.303933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.647911] LNet: 124710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2379.534545] Key type .llcrypt unregistered [ 2379.540219] Key type ._llcrypt unregistered [ 2386.680678] Key type ._llcrypt registered [ 2386.688299] Key type .llcrypt registered [ 2387.007122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2387.037365] alg: No test for adler32 (adler32-zlib) [ 2388.002341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2388.064698] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2397.737728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.112@tcp [ 2411.265359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2411.339207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2411.380778] LNet: Added LNI 192.168.204.12@tcp1 [8/256/0/180] [ 2412.556260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2413.478360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.112@tcp [ 2413.665143] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.112@tcp [ 2413.782991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2413.830828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2413.868868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2413.905592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2413.989178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2414.061767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2414.115298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2414.160678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2414.214889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2414.240815] LNet: 125364:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2414.266689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2414.302969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2414.338991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2414.373417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2414.439920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2414.501852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2415.693603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2416.255854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2416.348627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2416.410712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2416.476047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2416.553922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2416.610414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2416.688923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2416.781759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2416.844758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2416.880715] LNet: 125483:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2416.924519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2417.003907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2417.059478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2417.125904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2417.207428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2417.323816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2418.895629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.112@tcp [ 2419.518397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2419.620083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2419.684036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2419.753581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2419.816311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2419.878559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2419.960255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2420.033940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2420.202580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2420.270248] LNet: 125603:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2420.333798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2420.430620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2420.519394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2420.664705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2420.764410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2420.864541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2422.022700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2422.381422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.417973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2422.454170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2422.512295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2422.597223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2422.652418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2422.691196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2422.737264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2422.802790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2422.829373] LNet: 125722:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2422.854452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2422.913406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2422.977332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2423.042328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2423.103840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2423.191696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2429.337995] LNet: 125803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.345456] LNetError: 125803:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.349114] LNet: Removed LNI 192.168.204.12@tcp [ 2429.360271] LNet: Removed LNI 192.168.204.12@tcp1 [ 2429.775426] Key type .llcrypt unregistered [ 2429.784138] Key type ._llcrypt unregistered [ 2450.231603] Key type ._llcrypt registered [ 2450.234960] Key type .llcrypt registered [ 2451.644958] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 08:13:31 (1788524011) [ 2452.014617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2452.033258] alg: No test for adler32 (adler32-zlib) [ 2453.058828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2453.178227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2453.215093] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2454.101880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2454.142570] LNet: Added LNI 192.168.204.12@tcp2 [8/256/0/180] [ 2455.243272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.12@tcp [ 2455.325652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2456.351424] LNet: Removed LNI 192.168.204.12@tcp [ 2463.719683] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 08:13:42 (1788524022) [ 2465.558751] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2467.937295] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 08:13:47 (1788524027) [ 2475.798666] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 08:13:55 (1788524035) [ 2483.777423] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 08:14:03 (1788524043) [ 2491.451681] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 08:14:11 (1788524051) [ 2499.206935] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 08:14:18 (1788524058) [ 2507.115301] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 08:14:26 (1788524066) [ 2515.600365] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 08:14:35 (1788524075) [ 2524.793058] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 08:14:43 (1788524083) [ 2533.334327] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 08:14:52 (1788524092) [ 2542.134668] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 08:15:01 (1788524101) [ 2550.552601] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 08:15:09 (1788524109) [ 2559.031729] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 08:15:18 (1788524118) [ 2566.434223] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 08:15:25 (1788524125) [ 2573.747297] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 08:15:33 (1788524133) [ 2580.730723] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 08:15:40 (1788524140) [ 2589.416438] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 08:15:48 (1788524148) [ 2597.662516] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 08:15:56 (1788524156) [ 2605.033215] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 08:16:04 (1788524164) [ 2605.264856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2605.318279] LNet: 137500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2605.329158] LNetError: 137500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2605.350419] LNet: Removed LNI 192.168.204.12@tcp2 [ 2605.402748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2605.652500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2605.706035] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2606.924331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2606.950816] LNet: 137642:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2606.963872] LNetError: 137642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2606.984659] LNet: Removed LNI 192.168.204.12@tcp [ 2607.003371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2607.097586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2607.144304] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2615.541223] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 08:16:14 (1788524174) [ 2622.656382] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 08:16:22 (1788524182) [ 2630.757653] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 08:16:30 (1788524190) [ 2639.444880] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 08:16:39 (1788524199) [ 2648.611892] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 08:16:47 (1788524207) [ 2655.881208] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 08:16:55 (1788524215) [ 2662.163592] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 08:17:01 (1788524221) [ 2669.860709] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 08:17:08 (1788524228) [ 2679.765467] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 08:17:18 (1788524238) [ 2688.331165] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 08:17:27 (1788524247) [ 2696.770336] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 08:17:36 (1788524256) [ 2705.266183] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 08:17:44 (1788524264) [ 2713.456155] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2436 sec ========== 08:17:53 (1788524273) [ 2713.781857] LNet: 145111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2713.787165] LNetError: 145111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2713.794457] LNet: Removed LNI 192.168.204.12@tcp [ 2714.740489] Key type .llcrypt unregistered [ 2714.745731] Key type ._llcrypt unregistered [ 2727.968282] Key type ._llcrypt registered [ 2727.994240] Key type .llcrypt registered [ 2728.211919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2728.224543] alg: No test for adler32 (adler32-zlib) [ 2729.412272] Lustre: Lustre: Build Version: 2.17.58_2_gdacfdd2 [ 2729.744446] LNet: Added LNI 192.168.204.12@tcp [8/256/0/180] [ 2731.586095] Key type lgssc registered [ 2733.334701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2805.825260] Lustre: Mounted lustre-client - version 2.17.58_2_gdacfdd2 [ 2811.599070] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2820.132235] Lustre: Unmounted lustre-client [ 2859.677615] Key type lgssc unregistered [ 2859.999417] LNet: 148063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2860.004491] LNetError: 148063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2860.024910] LNet: Removed LNI 192.168.204.12@tcp [ 2860.892192] Key type .llcrypt unregistered [ 2860.893946] Key type ._llcrypt unregistered