[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 483148193 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002314] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008015] pid_max: default: 32768 minimum: 301 [ 0.009137] LSM: Security Framework initializing [ 0.010062] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.013065] *** VALIDATE selinux *** [ 0.021261] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026064] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027215] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] *** VALIDATE tmpfs *** [ 0.032393] *** VALIDATE proc *** [ 0.033224] *** VALIDATE cgroup *** [ 0.034021] *** VALIDATE cgroup2 *** [ 0.035265] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042631] debug: unmapping init [mem 0xffffffff97259000-0xffffffff97260fff] [ 0.044169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045709] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048016] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.054233] rcu: Hierarchical SRCU implementation. [ 0.056489] smp: Bringing up secondary CPUs ... [ 0.057554] x86: Booting SMP configuration: [ 0.058028] .... node #0, CPUs: #1 #2 #3 [ 0.062207] smp: Brought up 1 node, 4 CPUs [ 0.064025] smpboot: Max logical packages: 1 [ 0.065019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.237045] node 0 deferred pages initialised in 169ms [ 0.242132] devtmpfs: initialized [ 0.243466] x86/mm: Memory block size: 128MB [ 0.247111] gcov: version magic: 0x41383552 [ 0.250338] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.254155] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256451] pinctrl core: initialized pinctrl subsystem [ 0.258320] [ 0.258898] ************************************************************* [ 0.261025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263016] ** ** [ 0.266019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.268016] ** ** [ 0.271022] ** This means that this kernel is built to expose internal ** [ 0.273019] ** IOMMU data structures, which may compromise security on ** [ 0.275019] ** your system. ** [ 0.278022] ** ** [ 0.280018] ** If you see this message and you are not debugging the ** [ 0.282020] ** kernel, report this immediately to your vendor! ** [ 0.285021] ** ** [ 0.287016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.290022] ************************************************************* [ 0.292798] NET: Registered protocol family 16 [ 0.294545] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.297096] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.300086] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.303553] cpuidle: using governor menu [ 0.304751] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.307355] PCI: Using configuration type 1 for base access [ 0.309134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.319071] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.321047] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.325136] cryptd: max_cpu_qlen set to 1000 [ 0.328331] ACPI: Added _OSI(Module Device) [ 0.330061] ACPI: Added _OSI(Processor Device) [ 0.332018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.333012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.337644] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.343518] ACPI: Interpreter enabled [ 0.345095] ACPI: PM: (supports S0 S3 S4 S5) [ 0.347030] ACPI: Using IOAPIC for interrupt routing [ 0.349214] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.352470] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.364147] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.366047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.368020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.371078] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.375190] acpiphp: Slot [2] registered [ 0.377188] acpiphp: Slot [3] registered [ 0.378129] acpiphp: Slot [4] registered [ 0.380158] acpiphp: Slot [5] registered [ 0.382173] acpiphp: Slot [6] registered [ 0.383157] acpiphp: Slot [7] registered [ 0.385132] acpiphp: Slot [8] registered [ 0.386118] acpiphp: Slot [9] registered [ 0.387143] acpiphp: Slot [10] registered [ 0.389132] acpiphp: Slot [11] registered [ 0.390125] acpiphp: Slot [12] registered [ 0.392141] acpiphp: Slot [13] registered [ 0.393124] acpiphp: Slot [14] registered [ 0.395135] acpiphp: Slot [15] registered [ 0.396171] acpiphp: Slot [16] registered [ 0.398130] acpiphp: Slot [17] registered [ 0.400141] acpiphp: Slot [18] registered [ 0.401172] acpiphp: Slot [19] registered [ 0.403128] acpiphp: Slot [20] registered [ 0.404125] acpiphp: Slot [21] registered [ 0.406125] acpiphp: Slot [22] registered [ 0.408152] acpiphp: Slot [23] registered [ 0.409108] acpiphp: Slot [24] registered [ 0.411098] acpiphp: Slot [25] registered [ 0.412121] acpiphp: Slot [26] registered [ 0.414118] acpiphp: Slot [27] registered [ 0.415120] acpiphp: Slot [28] registered [ 0.417100] acpiphp: Slot [29] registered [ 0.418111] acpiphp: Slot [30] registered [ 0.420099] acpiphp: Slot [31] registered [ 0.421053] PCI host bridge to bus 0000:00 [ 0.422021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.424021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.426033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.429042] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.431029] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.434039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.436205] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.440735] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.444845] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.451982] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.456073] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.458029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.461027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.463027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.465524] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.466000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.468054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.471694] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.475015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.484958] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.490014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.494576] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.503014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.509018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.520016] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.529445] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.535022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.541027] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.558023] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.567324] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.570378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.572414] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.575376] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.577264] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.582129] iommu: Default domain type: Passthrough [ 0.584491] SCSI subsystem initialized [ 0.585106] ACPI: bus type USB registered [ 0.586106] usbcore: registered new interface driver usbfs [ 0.588070] usbcore: registered new interface driver hub [ 0.589064] usbcore: registered new device driver usb [ 0.591140] pps_core: LinuxPPS API ver. 1 registered [ 0.593011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.596058] PTP clock support registered [ 0.598075] EDAC MC: Ver: 3.0.0 [ 0.600152] PCI: Using ACPI for IRQ routing [ 0.602877] NetLabel: Initializing [ 0.604016] NetLabel: domain hash size = 128 [ 0.605011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.606110] NetLabel: unlabeled traffic allowed by default [ 0.609173] vgaarb: loaded [ 0.610308] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.612022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.618434] clocksource: Switched to clocksource kvm-clock [ 0.725626] VFS: Disk quotas dquot_6.6.0 [ 0.727388] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.730822] *** VALIDATE ramfs *** [ 0.732190] *** VALIDATE hugetlbfs *** [ 0.733735] pnp: PnP ACPI init [ 0.736184] pnp: PnP ACPI: found 6 devices [ 0.758202] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.761345] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.763539] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.765446] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.768129] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.770637] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.773507] NET: Registered protocol family 2 [ 0.776245] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.781421] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.784815] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.789198] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.792498] TCP: Hash tables configured (established 65536 bind 65536) [ 0.795575] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.798152] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.801144] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.804330] NET: Registered protocol family 1 [ 0.806466] RPC: Registered named UNIX socket transport module. [ 0.808884] RPC: Registered udp transport module. [ 0.810710] RPC: Registered tcp transport module. [ 0.812269] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.814740] NET: Registered protocol family 44 [ 0.816508] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.818806] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.821044] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.823370] PCI: CLS 0 bytes, default 64 [ 0.825182] Unpacking initramfs... [ 2.246797] debug: unmapping init [mem 0xffff8b43fcc64000-0xffff8b43fffcffff] [ 2.250746] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.253095] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.255808] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.757903] Initialise system trusted keyrings [ 2.759627] Key type blacklist registered [ 2.762083] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.774927] zbud: loaded [ 2.780319] *** VALIDATE nfs *** [ 2.781654] *** VALIDATE nfs4 *** [ 2.783263] pstore: using deflate compression [ 2.786303] Platform Keyring initialized [ 2.908326] NET: Registered protocol family 38 [ 2.910373] Key type asymmetric registered [ 2.912009] Asymmetric key parser 'x509' registered [ 2.914089] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.917576] io scheduler mq-deadline registered [ 2.918996] io scheduler kyber registered [ 2.920699] io scheduler bfq registered [ 2.922729] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.925623] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.928926] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.931561] ACPI: Power Button [PWRF] [ 3.022136] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.119496] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.218968] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.253097] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.286294] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.291768] Non-volatile memory driver v1.3 [ 3.293338] Linux agpgart interface v0.103 [ 3.328764] virtio_blk virtio1: [vda] 133856 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.332610] vda: detected capacity change from 0 to 68534272 [ 3.347339] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.350643] vdb: detected capacity change from 0 to 1073741824 [ 3.357963] libphy: Fixed MDIO Bus: probed [ 3.367476] usbcore: registered new interface driver usbserial_generic [ 3.370279] usbserial: USB Serial support registered for generic [ 3.373538] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.377837] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.379826] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.382614] mousedev: PS/2 mouse device common for all mice [ 3.385635] rtc_cmos 00:05: RTC can wake from S4 [ 3.388597] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.392547] rtc_cmos 00:05: registered as rtc0 [ 3.395493] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.395764] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.402183] intel_pstate: CPU model not supported [ 3.402278] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.408296] hid: raw HID events driver (C) Jiri Kosina [ 3.410490] usbcore: registered new interface driver usbhid [ 3.412375] usbhid: USB HID core driver [ 3.413971] drop_monitor: Initializing network drop monitor service [ 3.416419] Initializing XFRM netlink socket [ 3.418730] NET: Registered protocol family 10 [ 3.421988] Segment Routing with IPv6 [ 3.423467] NET: Registered protocol family 17 [ 3.426133] mpls_gso: MPLS GSO support [ 3.434101] RAS: Correctable Errors collector initialized. [ 3.436888] AVX version of gcm_enc/dec engaged. [ 3.438451] AES CTR mode by8 optimization enabled [ 3.519302] sched_clock: Marking stable (3519275466, 0)->(4423588927, -904313461) [ 3.523266] registered taskstats version 1 [ 3.525287] Loading compiled-in X.509 certificates [ 3.527773] zswap: loaded using pool lzo/zbud [ 3.556592] Key type big_key registered [ 3.571493] Key type encrypted registered [ 3.573401] ima: No TPM chip found, activating TPM-bypass! [ 3.575827] ima: Allocated hash algorithm: sha1 [ 3.577966] ima: No architecture policies found [ 3.580051] evm: Initialising EVM extended attributes: [ 3.582012] evm: security.selinux [ 3.582975] evm: security.ima [ 3.584183] evm: security.capability [ 3.585660] evm: HMAC attrs: 0x1 [ 3.588150] rtc_cmos 00:05: setting system clock to 2025-11-18 23:04:30 UTC (1763507070) [ 3.596436] debug: unmapping init [mem 0xffffffff98203000-0xffffffff983fffff] [ 3.600310] debug: unmapping init [mem 0xffffffff96f82000-0xffffffff97258fff] [ 3.614125] Write protecting the kernel read-only data: 28672k [ 3.617750] debug: unmapping init [mem 0xffffffff95603000-0xffffffff957fffff] [ 3.620522] debug: unmapping init [mem 0xffffffff95f14000-0xffffffff95ffffff] [ 3.658162] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.666156] systemd[1]: Detected virtualization kvm. [ 3.668622] systemd[1]: Detected architecture x86-64. [ 3.670679] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.696968] systemd[1]: No hostname configured. [ 3.698905] systemd[1]: Set hostname to . [ 3.700851] random: systemd: uninitialized urandom read (16 bytes read) [ 3.703369] systemd[1]: Initializing machine ID from random generator. [ 3.832146] random: systemd: uninitialized urandom read (16 bytes read) [ 3.835289] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.840775] random: systemd: uninitialized urandom read (16 bytes read) [ 3.844084] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.849349] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.503160] device-mapper: uevent: version 1.0.3 [ 4.505386] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 5.169764] virtio_net virtio0 ens2: renamed from eth0 Started Hardware RNG Entropy Gatherer Daemon.[ 5.179294] random: fast init done [ 5.212195] scsi host0: ata_piix [ 5.250361] scsi host1: ata_piix [ 5.323901] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.326685] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.939923] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.091661] random: crng init done [ 10.092682] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.519909] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.666961] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.925743] SELinux: Disabled at runtime. [ 11.992273] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.001590] systemd[1]: Detected virtualization kvm. [ 12.003534] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.515519] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.519197] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.527888] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.531987] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.535588] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.544186] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.549201] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Slices. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [FAILED] Failed to set up automount [ 12.637753] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.026685] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.307581] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.447880] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.484940] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.514095] EDAC sbridge: Ver: 1.1.2 [ 14.622920] Key type dns_resolver registered [ 14.927857] NFS: Registering the id_resolver key type [ 14.929997] Key type id_resolver registered [ 14.931563] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg420-client login: [ 36.726597] hrtimer: interrupt took 5481413 ns [ 60.558619] libcfs: loading out-of-tree module taints kernel. [ 60.667979] Key type ._llcrypt registered [ 60.670024] Key type .llcrypt registered [ 61.472795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 61.487438] alg: No test for adler32 (adler32-zlib) [ 62.731695] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 63.605446] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 65.329914] Key type lgssc registered [ 67.133568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 169.973771] Lustre: Mounted lustre-client [ 174.838633] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 190.608891] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing check_logdir /tmp/testlogs/ [ 194.336880] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing yml_node [ 195.589724] Lustre: lustre-OST0000-osc-ffff8b4450d88800: disconnect after 23s idle [ 199.087704] Lustre: DEBUG MARKER: Client: 2.16.61.43 [ 201.648335] Lustre: DEBUG MARKER: MDS: 2.16.61.43 [ 204.393956] Lustre: DEBUG MARKER: OSS: 2.16.61.43 [ 205.807543] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Nov 18 18:07:51 EST 2025 [ 220.367920] LustreError: 7917:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8b4450d88800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 220.380974] LustreError: 7917:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 220.414166] Lustre: Unmounted lustre-client [ 240.815608] Key type lgssc unregistered [ 241.041730] LNet: 8363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 241.055393] LNetError: 8363:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 242.087211] LNet: Removed LNI 192.168.204.20@tcp [ 242.604566] Key type .llcrypt unregistered [ 242.607446] Key type ._llcrypt unregistered [ 263.383997] Key type ._llcrypt registered [ 263.395639] Key type .llcrypt registered [ 263.625740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 263.647430] alg: No test for adler32 (adler32-zlib) [ 264.524719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 264.587366] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 264.607573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 268.199834] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 268.360586] LNet: 9234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 268.371453] LNetError: 9234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 268.409274] LNet: Removed LNI 192.168.204.20@tcp [ 268.958826] Key type .llcrypt unregistered [ 268.960540] Key type ._llcrypt unregistered [ 273.980546] Key type ._llcrypt registered [ 273.984230] Key type .llcrypt registered [ 275.171351] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:09:00 (1763507340) [ 275.445218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 275.466505] alg: No test for adler32 (adler32-zlib) [ 276.350579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.430465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.449083] LNet: 9643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.469795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.539517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 276.559343] LNet: 9650:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 282.317939] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:09:08 (1763507348) [ 282.383156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.410523] LNet: 10220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.436649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.532756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 288.515791] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:09:14 (1763507354) [ 288.579787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.597224] LNet: 10805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.627300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.716745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 295.012227] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:09:20 (1763507360) [ 295.119373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.159293] LNet: 11391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.198316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.314324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 301.601152] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:09:26 (1763507366) [ 301.707227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.754622] LNet: 11975:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.786302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.925773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 302.563425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 309.696875] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:09:35 (1763507375) [ 309.811402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.847535] LNet: 12565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.866818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.963878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 310.509276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 311.002360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 317.041489] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:09:42 (1763507382) [ 317.157953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.213716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.306685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 324.150655] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 18:09:49 (1763507389) [ 324.256318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.337571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.450763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 324.502433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 324.974933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 325.035880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 325.349422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 325.381775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 325.826209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 325.867500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 326.276496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 326.300602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 326.641538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 326.678404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 327.111202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 327.158127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 327.594101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 327.624919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 328.201581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 328.238310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 328.545519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 328.580872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 328.874372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 328.903533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 329.219939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 329.265421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 329.626640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 329.711499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 336.804360] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 18:10:02 (1763507402) [ 336.896161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.934208] LNet: 14425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.940838] LNet: 14425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 336.982858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.072921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 337.098871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 344.080749] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 18:10:09 (1763507409) [ 344.242956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.329710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.382457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 344.416929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 350.758676] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 18:10:16 (1763507416) [ 350.954521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.985842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.076308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 351.122985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 357.848864] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 18:10:23 (1763507423) [ 357.978138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.027748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.098267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 358.149633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 364.904790] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 18:10:30 (1763507430) [ 365.172778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.222845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.314593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 365.349748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 371.679550] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 18:10:36 (1763507436) [ 371.770214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.806418] LNet: 17358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.815102] LNet: 17358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 371.858525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.980949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 372.035023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 378.678299] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 18:10:44 (1763507444) [ 378.837239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.894439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.004940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.350668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 386.137360] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 18:10:51 (1763507451) [ 386.224877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.292246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.499207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 386.768563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 393.615159] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 18:10:59 (1763507459) [ 393.695571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.778405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.950841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 394.185539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 400.382141] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 18:11:06 (1763507466) [ 400.581044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.671749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.179724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.519823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 407.443378] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 18:11:13 (1763507473) [ 407.543053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.620697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.745407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 407.779327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 413.417513] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 18:11:19 (1763507479) [ 413.515717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.573347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.637960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 413.691934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.162429] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 18:11:25 (1763507485) [ 419.282744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.321421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.391751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 424.530066] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 18:11:30 (1763507490) [ 424.601917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.627227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.695989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 429.610491] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 18:11:35 (1763507495) [ 429.696754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.733266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.843976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 430.103767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 434.869834] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 18:11:40 (1763507500) [ 435.008541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.112763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.238033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 435.273401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 441.018693] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 18:11:46 (1763507506) [ 441.084343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.107517] LNet: 23971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.110173] LNet: 23971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 441.126855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.204685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 441.230642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 446.221582] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 18:11:52 (1763507512) [ 446.292943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.312894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.391725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 446.422734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 451.737794] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 18:11:57 (1763507517) [ 451.974876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.024932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.131136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 452.165425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 457.627521] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 18:12:03 (1763507523) [ 457.730794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.768655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.852196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 457.920047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 457.939034] LNetError: 25748:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 457.980320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 458.007911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 462.802631] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 18:12:08 (1763507528) [ 462.886715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.927066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.993686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GHCkktLuRw [ 463.048193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GHCkktLuRw [ 463.088827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.GHCkktLuRw [ 467.713330] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 18:12:13 (1763507533) [ 467.791198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.843893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.890209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 467.913820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 467.947078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 472.755989] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 18:12:18 (1763507538) [ 472.891036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.967979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.041657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 475.651295] LNet: Message forwarding will be enabled in 2s [ 478.752145] LNet: Message forwarding enabled [ 479.165612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 479.178216] LNet: Message forwarding will stop in 2s to 6s [ 483.575952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 483.957649] LNet: Message forwarding disabled [ 485.812918] LNet: Message forwarding will be enabled in 2s [ 488.101170] LNet: Message forwarding enabled [ 493.832494] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 18:12:39 (1763507559) [ 493.953945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.162344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.229487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 502.044360] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 18:12:47 (1763507567) [ 502.128760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.173643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.284169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 502.339065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 502.372644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 502.396758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 502.437821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 502.480096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 502.525409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 502.551996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 502.580931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 502.623470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 502.651537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 502.698643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 502.751573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 502.788594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 502.854466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 502.885659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 502.962206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 503.000661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 503.064331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 503.096250] LNetError: 28817:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 509.226375] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 18:12:54 (1763507574) [ 509.300199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.336888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.404416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 515.901053] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 18:13:01 (1763507581) [ 517.641640] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg420-server [ 519.344472] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 18:13:04 (1763507584) [ 519.583551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.642643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.690957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 519.723772] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 521.429792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[21-25]@tcp [ 521.454544] LNet: 30533:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 521.526744] LNetError: 30322:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.23@tcp is being used as a gateway but routing feature is not turned on [ 521.546669] LNetError: 30322:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.23@tcp has gone from up to down [ 522.542095] LNetError: 30322:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.23@tcp is being used as a gateway but routing feature is not turned on [ 522.551724] LNetError: 30322:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 523.566627] LNetError: 30322:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.24@tcp is being used as a gateway but routing feature is not turned on [ 523.582719] LNetError: 30322:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 525.617103] LNetError: 30322:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.22@tcp is being used as a gateway but routing feature is not turned on [ 525.638393] LNetError: 30322:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 527.299588] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 18:13:13 (1763507593) [ 528.769792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg420-server [ 529.715568] LNetError: 30322:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.25@tcp is being used as a gateway but routing feature is not turned on [ 529.730677] LNetError: 30322:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 530.333285] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 18:13:16 (1763507596) [ 530.449429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.492095] LNetError: 31461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 530.546907] LNet: Removed LNI 192.168.204.20@tcp [ 530.578440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.670571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 530.713721] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 531.543058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[21-26/2]@tcp [ 531.576518] LNet: 31585:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 531.592403] LNet: 31585:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 531.636202] LNetError: 31469:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.23@tcp has gone from up to down [ 531.644692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[21-26/2]@tcp [ 531.651352] LNetError: 31469:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 538.091447] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 18:13:23 (1763507603) [ 538.238373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.264169] LNetError: 32177:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 538.280896] LNet: Removed LNI 192.168.204.20@tcp [ 538.296825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.360898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 538.418287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.448393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.511136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 538.578460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.624621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.694445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 538.857251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.936431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.035526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.119528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.147107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.197903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 539.263987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.324611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.378263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.498963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.555126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.627881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 539.723834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.748306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.842484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.991654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.061930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.202888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 540.297658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.355214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.457357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.546358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.592965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.653942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 540.716380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.788957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.883744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 546.860506] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 18:13:32 (1763507612) [ 548.664707] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg420-server [ 550.594248] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 18:13:36 (1763507616) [ 552.634832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg420-server [ 554.421638] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 18:13:39 (1763507619) [ 555.376232] Key type .llcrypt unregistered [ 555.377984] Key type ._llcrypt unregistered [ 560.669600] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 560.776288] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 560.821054] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 560.833281] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 563.469107] Key type ._llcrypt registered [ 563.470693] Key type .llcrypt registered [ 563.691290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 563.710098] alg: No test for adler32 (adler32-zlib) [ 564.681736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.762319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 564.799741] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 565.465961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 565.485866] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 567.175877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 567.192491] LNet: Removed LNI 10.1.2.3@tcp [ 567.288030] LNet: 34295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.293694] LNetError: 34295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.305163] LNet: Removed LNI 192.168.204.20@tcp [ 567.843523] Key type .llcrypt unregistered [ 567.850847] Key type ._llcrypt unregistered [ 575.788145] Key type ._llcrypt registered [ 575.791829] Key type .llcrypt registered [ 576.918975] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 18:14:02 (1763507642) [ 577.329275] Key type .llcrypt unregistered [ 577.331660] Key type ._llcrypt unregistered [ 581.535145] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 581.691849] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 581.696444] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 581.703725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 584.118125] Key type ._llcrypt registered [ 584.119332] Key type .llcrypt registered [ 584.292354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 584.304081] alg: No test for adler32 (adler32-zlib) [ 585.186605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.227694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.252878] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 586.017159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 586.040747] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 588.263083] LNet: 35780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.277279] LNetError: 35780:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.281990] LNet: Removed LNI 192.168.204.20@tcp [ 588.297944] LNet: Removed LNI 10.1.2.3@tcp [ 588.844751] Key type .llcrypt unregistered [ 588.848461] Key type ._llcrypt unregistered [ 596.790292] Key type ._llcrypt registered [ 596.792353] Key type .llcrypt registered [ 597.734172] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 18:14:23 (1763507663) [ 598.077469] Key type .llcrypt unregistered [ 598.079996] Key type ._llcrypt unregistered [ 601.899500] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 602.062662] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 602.065495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 602.067881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 604.583369] Key type ._llcrypt registered [ 604.591137] Key type .llcrypt registered [ 604.743741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 604.759351] alg: No test for adler32 (adler32-zlib) [ 605.708286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.766219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 605.806434] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 607.772057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 607.806140] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 610.019368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 610.031077] LNet: Removed LNI 10.1.2.3@tcp [ 610.062559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 610.090033] LNet: Removed LNI 10.1.2.31@tcp [ 610.100838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 610.126746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 610.784726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 610.805143] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 611.793605] LNet: 37691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.806597] LNetError: 37691:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 611.809734] LNet: Removed LNI 10.1.2.3@tcp [ 611.824594] LNet: Removed LNI 10.1.2.31@tcp [ 612.366532] Key type .llcrypt unregistered [ 612.368389] Key type ._llcrypt unregistered [ 622.094073] Key type ._llcrypt registered [ 622.096155] Key type .llcrypt registered [ 623.608980] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 18:14:48 (1763507688) [ 624.286996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.317817] alg: No test for adler32 (adler32-zlib) [ 625.231823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.275454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 625.304632] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 626.166977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.189811] LNet: 38640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.194088] LNetError: 38640:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.217280] LNet: Removed LNI 192.168.204.20@tcp [ 626.238672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.283925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.20@tcp [ 626.308350] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 627.028894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.20@tcp [ 627.059543] LNet: Removed LNI 192.168.204.20@tcp [ 627.135344] LNet: 38775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.673426] Key type .llcrypt unregistered [ 627.678028] Key type ._llcrypt unregistered [ 637.212952] Key type ._llcrypt registered [ 637.214846] Key type .llcrypt registered [ 638.476093] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 18:15:04 (1763507704) [ 640.282655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg420-server [ 642.007574] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 18:15:07 (1763507707) [ 642.447610] Key type .llcrypt unregistered [ 642.449360] Key type ._llcrypt unregistered [ 647.049442] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 647.235534] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 647.255572] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 647.268992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 649.830464] Key type ._llcrypt registered [ 649.831862] Key type .llcrypt registered [ 650.093903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 650.126881] alg: No test for adler32 (adler32-zlib) [ 651.141578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.221652] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 652.134028] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 654.417756] LNet: 40604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.425492] LNetError: 40604:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.436891] LNet: Removed LNI 192.168.204.20@tcp1 [ 654.444848] LNet: Removed LNI 10.1.2.3@tcp2 [ 655.095696] Key type .llcrypt unregistered [ 655.099137] Key type ._llcrypt unregistered [ 666.915238] Key type ._llcrypt registered [ 666.921841] Key type .llcrypt registered [ 668.467774] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 18:15:33 (1763507733) [ 668.945447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 668.965633] alg: No test for adler32 (adler32-zlib) [ 669.942807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.998456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 670.051762] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 671.154441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.187066] LNet: 41570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.192838] LNetError: 41570:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.212110] LNet: Removed LNI 192.168.204.20@tcp [ 671.235884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.347845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 671.424570] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 678.905358] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 18:15:44 (1763507744) [ 678.998546] LNet: 42155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.015081] LNetError: 42155:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.035722] LNet: Removed LNI 192.168.204.20@tcp [ 679.611487] Key type .llcrypt unregistered [ 679.614261] Key type ._llcrypt unregistered [ 684.303521] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 684.348174] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 684.352377] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 684.355671] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 686.890632] Key type ._llcrypt registered [ 686.892493] Key type .llcrypt registered [ 687.079781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 687.102963] alg: No test for adler32 (adler32-zlib) [ 688.058480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.131521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.170344] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 689.018880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 689.030875] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 691.026950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.042049] LNet: 42904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.053242] LNetError: 42904:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.056505] LNet: Removed LNI 192.168.204.20@tcp [ 691.067230] LNet: Removed LNI 10.1.2.3@tcp [ 691.089668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.136941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 691.161527] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 691.164948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 691.478649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.504331] LNet: Removed LNI 192.168.204.20@tcp [ 691.516804] LNet: Removed LNI 10.1.2.3@tcp [ 691.547263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.599462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 691.637661] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 691.643480] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 692.035153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 692.051478] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 693.142835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.168445] LNet: 43093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.173601] LNet: 43093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 693.180731] LNetError: 43093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.185380] LNetError: 43093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 693.190377] LNet: Removed LNI 192.168.204.20@tcp [ 693.192253] LNet: Removed LNI 10.1.2.3@tcp [ 693.210667] LNet: Removed LNI 192.168.204.20@tcp2 [ 693.237841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 693.319871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 693.347968] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 693.350727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 693.356591] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 700.277252] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 18:16:05 (1763507765) [ 700.337939] LNet: 43684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.344097] LNetError: 43684:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.351483] LNetError: 43684:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 700.355738] LNet: Removed LNI 192.168.204.20@tcp [ 700.364337] LNet: Removed LNI 10.1.2.3@tcp [ 700.396226] LNet: Removed LNI 192.168.204.20@tcp2 [ 701.128975] Key type .llcrypt unregistered [ 701.131930] Key type ._llcrypt unregistered [ 705.767151] Key type ._llcrypt registered [ 705.768789] Key type .llcrypt registered [ 705.976860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 705.991232] alg: No test for adler32 (adler32-zlib) [ 706.903996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.995381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 707.018617] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 707.821706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.848856] LNet: 44083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.862781] LNetError: 44083:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.873345] LNet: Removed LNI 192.168.204.20@tcp [ 707.906508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.982270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 708.010060] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 708.413486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.435504] LNet: 44113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.439700] LNetError: 44113:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 708.456266] LNet: Removed LNI 192.168.204.20@tcp [ 708.472525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.521517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 708.548835] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 708.982899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.011095] LNet: Removed LNI 192.168.204.20@tcp [ 709.036236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.121848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 709.155323] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 709.533850] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 709.650483] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 709.652475] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 709.660445] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 712.014205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.041750] LNet: 44220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.048589] LNet: 44220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 712.054964] LNetError: 44220:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.060771] LNetError: 44220:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 712.075819] LNet: Removed LNI 192.168.204.20@tcp [ 712.096518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.174590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 712.198932] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 718.331227] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 18:16:23 (1763507783) [ 718.416082] LNet: 44809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.425037] LNetError: 44809:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.436428] LNet: Removed LNI 192.168.204.20@tcp [ 718.984687] Key type .llcrypt unregistered [ 718.986210] Key type ._llcrypt unregistered [ 723.828907] Key type ._llcrypt registered [ 723.833315] Key type .llcrypt registered [ 724.130673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.151517] alg: No test for adler32 (adler32-zlib) [ 725.051777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.094832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.143888] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 725.950639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 725.967705] LNet: 45211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.974490] LNetError: 45211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.987756] LNet: Removed LNI 192.168.204.20@tcp [ 726.011721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.068476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 726.093648] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 732.001042] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:16:37 (1763507797) [ 732.221715] LNet: 45804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.228217] LNetError: 45804:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 732.241920] LNet: Removed LNI 192.168.204.20@tcp [ 732.845651] Key type .llcrypt unregistered [ 732.850703] Key type ._llcrypt unregistered [ 738.221207] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 738.372053] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 738.374696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 738.378038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 740.824858] Key type ._llcrypt registered [ 740.826431] Key type .llcrypt registered [ 740.970545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.979600] alg: No test for adler32 (adler32-zlib) [ 741.891236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.966614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.005434] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 742.771876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.793117] LNet: 46248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.799180] LNetError: 46248:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.813891] LNet: Removed LNI 192.168.204.20@tcp [ 742.840274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.911060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 742.932300] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 748.568209] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 18:16:54 (1763507814) [ 748.644391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.666933] LNet: 46838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.672150] LNetError: 46838:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.683078] LNet: Removed LNI 192.168.204.20@tcp [ 748.713859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.809655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 753.873773] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 18:16:59 (1763507819) [ 753.941532] LNet: 47419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.448574] Key type .llcrypt unregistered [ 754.454281] Key type ._llcrypt unregistered [ 759.121916] Key type ._llcrypt registered [ 759.123552] Key type .llcrypt registered [ 759.379772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.407449] alg: No test for adler32 (adler32-zlib) [ 760.289825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.332256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 760.361281] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 761.111744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.134696] LNet: 47821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.141302] LNetError: 47821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.151171] LNet: Removed LNI 192.168.204.20@tcp [ 761.165384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.205830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 761.238327] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 766.780582] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 18:17:12 (1763507832) [ 766.932254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 766.954415] LNet: 48414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.960721] LNetError: 48414:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 766.970804] LNet: Removed LNI 192.168.204.20@tcp [ 766.990810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.051433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 771.568515] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 18:17:17 (1763507837) [ 772.711445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 774.087443] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 18:17:19 (1763507839) [ 775.240921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 776.552370] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 18:17:22 (1763507842) [ 778.003797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 779.406254] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 18:17:25 (1763507845) [ 780.751816] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 782.222958] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 18:17:27 (1763507847) [ 783.424606] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 784.731238] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 18:17:30 (1763507850) [ 784.846877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.878870] LNet: 50464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.899424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.946335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.971687] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 785.617112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.20 [ 785.628796] LNet: Removed LNI 192.168.204.20@tcp [ 790.360025] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 18:17:36 (1763507856) [ 790.490923] LNet: 51167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.499771] LNetError: 51167:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.530623] LNet: Removed LNI 192.168.204.20@tcp [ 790.942553] Key type .llcrypt unregistered [ 790.944341] Key type ._llcrypt unregistered [ 795.306324] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.518535] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.525731] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.530530] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 798.114689] Key type ._llcrypt registered [ 798.116010] Key type .llcrypt registered [ 798.263195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.273690] alg: No test for adler32 (adler32-zlib) [ 799.586300] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.123486] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:17:50 (1763507870) [ 805.207426] LNet: 52059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.215747] LNetError: 52059:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 805.234833] LNet: Removed LNI 10.1.2.3@tcp [ 805.708401] Key type .llcrypt unregistered [ 805.710458] Key type ._llcrypt unregistered [ 809.979770] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 810.165988] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 810.169856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 810.183761] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 812.758826] Key type ._llcrypt registered [ 812.760365] Key type .llcrypt registered [ 812.882743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 812.894695] alg: No test for adler32 (adler32-zlib) [ 813.856207] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.368837] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:18:04 (1763507884) [ 818.546086] LNet: 52958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.557515] LNetError: 52958:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 818.578960] LNet: Removed LNI 10.1.2.3@tcp [ 819.124603] Key type .llcrypt unregistered [ 819.125984] Key type ._llcrypt unregistered [ 823.312437] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 823.473427] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 823.476751] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 823.482867] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 826.079983] Key type ._llcrypt registered [ 826.083849] Key type .llcrypt registered [ 826.217098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 826.228730] alg: No test for adler32 (adler32-zlib) [ 827.139725] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 831.419861] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:18:17 (1763507897) [ 831.616245] LNet: 53844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.635177] LNetError: 53844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 831.673039] LNet: Removed LNI 10.1.2.3@tcp [ 832.309255] Key type .llcrypt unregistered [ 832.311046] Key type ._llcrypt unregistered [ 836.379571] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 836.529861] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 836.532437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 836.534660] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 839.069803] Key type ._llcrypt registered [ 839.071724] Key type .llcrypt registered [ 839.220195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 839.230633] alg: No test for adler32 (adler32-zlib) [ 840.175771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.293413] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 847.165245] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:18:32 (1763507912) [ 847.654463] LNet: 55072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.659870] LNetError: 55072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 847.674265] LNet: Removed LNI 10.1.2.3@tcp [ 848.226747] Key type .llcrypt unregistered [ 848.228413] Key type ._llcrypt unregistered [ 853.105992] Key type ._llcrypt registered [ 853.108684] Key type .llcrypt registered [ 853.269910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.282838] alg: No test for adler32 (adler32-zlib) [ 854.154686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 854.180295] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 858.990202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 859.438902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_interrupt -r 1 [ 859.470237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_interrupt -r 1 [ 859.498148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 859.511845] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 859.615446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 859.647925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 859.679662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 859.704887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 859.736581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 859.768152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 859.813514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 859.860429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 859.892721] LNet: 55506:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 859.905989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 859.926144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 859.951948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 859.970638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 859.999631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 860.024479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 860.050745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 860.355596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_dropped -r 1 [ 860.393094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_dropped -r 1 [ 860.419369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 860.433587] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 860.602171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 860.628981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 860.656351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 860.679361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 860.710447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 860.733295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 860.762357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 860.786815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 860.808231] LNet: 55615:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 860.829317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 860.849780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 860.868375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 860.884886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 860.915811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 860.941150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 860.968219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 861.349702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_aborted -r 1 [ 861.376173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_aborted -r 1 [ 861.407821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 861.568373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 861.590116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 861.614235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 861.636241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 861.653597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 861.673528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 861.694647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 861.713221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 861.732641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 861.752542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 861.770304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 861.786139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 861.805722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 861.833354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 861.868433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 862.264821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_no_route -r 1 [ 862.289198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_no_route -r 1 [ 862.307952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 862.320646] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 862.323991] LNet: Skipped 1 previous similar message [ 862.368324] LNet: 1 local NIs in recovery (showing 1): 192.168.204.20@tcp [ 862.475884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 862.494309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 862.511884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 862.529211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 862.548699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 862.569896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 862.593620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 862.617039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 862.631941] LNet: 55833:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 862.636144] LNet: 55833:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 862.651268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 862.682702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 862.703568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 862.722636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 862.746240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 862.767909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 862.789860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 863.257553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_timeout -r 1 [ 863.298165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_timeout -r 1 [ 863.328767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 863.553522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 863.584209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 863.647817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 863.687402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 863.722289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 863.762414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 863.797369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 863.845053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 863.885214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 863.912650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 863.935591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 863.964559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 863.998200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 864.033271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 864.064362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 864.539841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_error -r 1 [ 864.571352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_error -r 1 [ 864.602626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 864.621808] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 864.635525] LNet: Skipped 2 previous similar messages [ 864.837153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 864.862446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 864.889360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 864.921318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 864.949028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 864.977980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 865.011651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 865.041203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 865.066146] LNet: 56052:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 865.069614] LNet: 56052:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 865.085917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 865.109234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 865.135899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 865.156082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 865.174688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 865.218309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 865.249189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 865.488174] LNet: 56092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.503402] LNetError: 56092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.520701] LNet: Removed LNI 192.168.204.20@tcp [ 865.881581] Key type .llcrypt unregistered [ 865.883358] Key type ._llcrypt unregistered [ 878.053216] Key type ._llcrypt registered [ 878.054770] Key type .llcrypt registered [ 879.046318] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:19:04 (1763507944) [ 879.752496] Key type .llcrypt unregistered [ 879.756357] Key type ._llcrypt unregistered [ 883.078432] Key type ._llcrypt registered [ 883.080138] Key type .llcrypt registered [ 883.203624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.216249] alg: No test for adler32 (adler32-zlib) [ 884.054269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 884.069687] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 888.533751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 895.151888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.180373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 895.196623] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 896.231204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_interrupt -r 1 [ 896.262701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_interrupt -r 1 [ 896.306922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_interrupt -r 1 [ 896.358960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_interrupt -r 1 [ 896.387622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_interrupt -r 1 [ 896.420763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_interrupt -r 1 [ 896.450679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_interrupt -r 1 [ 896.484520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_interrupt -r 1 [ 896.512079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 896.538654] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 896.545993] LNet: 1 local NIs in recovery (showing 1): 192.168.204.20@tcp [ 896.810531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 896.846744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 896.880614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 896.921539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 896.964065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 896.989024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 897.028747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 897.056114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 897.071359] LNet: 57550:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 897.087421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 897.115140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 897.135203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 897.156172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 897.187458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 897.224386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 897.261135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 897.805594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_dropped -r 1 [ 897.865865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_dropped -r 1 [ 897.907295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_dropped -r 1 [ 897.930570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_dropped -r 1 [ 897.967383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_dropped -r 1 [ 897.997685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_dropped -r 1 [ 898.024199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_dropped -r 1 [ 898.046812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_dropped -r 1 [ 898.078643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 898.092912] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 898.097761] LNet: Skipped 2 previous similar messages [ 898.249370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 898.267021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 898.286652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 898.313330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 898.341587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 898.358935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 898.394422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 898.426230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 898.441984] LNet: 57671:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 898.459994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 898.483915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 898.502703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 898.517705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 898.536159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 898.558261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 898.583490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 898.969037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_aborted -r 1 [ 898.994285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_aborted -r 1 [ 899.027685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_aborted -r 1 [ 899.045431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_aborted -r 1 [ 899.070364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_aborted -r 1 [ 899.092990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_aborted -r 1 [ 899.114448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_aborted -r 1 [ 899.148335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_aborted -r 1 [ 899.174675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 899.185633] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 899.188646] LNet: Skipped 2 previous similar messages [ 899.313910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 899.333810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 899.350744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 899.367499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 899.388390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 899.415544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 899.434945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 899.468505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 899.480581] LNet: 57793:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 899.492630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 899.519321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 899.536989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 899.571072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 899.608672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 899.635867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 899.659245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 899.975383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_no_route -r 1 [ 899.999689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_no_route -r 1 [ 900.018787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_no_route -r 1 [ 900.047735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_no_route -r 1 [ 900.077499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_no_route -r 1 [ 900.109666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_no_route -r 1 [ 900.151438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_no_route -r 1 [ 900.177940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_no_route -r 1 [ 900.211115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 900.366623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 900.386612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 900.421680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 900.444599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 900.472284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 900.490473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 900.514407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 900.549096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 900.574865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 900.616862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 900.651230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 900.675545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 900.707524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 900.739977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 900.775377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 901.115144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_timeout -r 1 [ 901.150255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_timeout -r 1 [ 901.173833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_timeout -r 1 [ 901.203718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_timeout -r 1 [ 901.234976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_timeout -r 1 [ 901.262116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_timeout -r 1 [ 901.282362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_timeout -r 1 [ 901.316849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_timeout -r 1 [ 901.351433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 901.368145] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 901.371443] LNet: Skipped 5 previous similar messages [ 901.576069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 901.597493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 901.640886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 901.687077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 901.722538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 901.757381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 901.783277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 901.807944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 901.826574] LNet: 58035:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 901.830219] LNet: 58035:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 901.847026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 901.863937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 901.884706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 901.906501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 901.934916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 901.963710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 901.996806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 902.385871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_error -r 1 [ 902.412963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_error -r 1 [ 902.438272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_error -r 1 [ 902.464617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_error -r 1 [ 902.487515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_error -r 1 [ 902.507770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_error -r 1 [ 902.533519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_error -r 1 [ 902.565518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_error -r 1 [ 902.597611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 902.815824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 902.846727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 902.882781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 902.907991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 902.944507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 902.974658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 903.005139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 903.042354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 903.064989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 903.104332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 903.127357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 903.151558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 903.175118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 903.206564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 903.234948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 906.701432] LNet: 58231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.714727] LNetError: 58231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.744157] LNet: Removed LNI 192.168.204.20@tcp [ 907.748989] LNetError: 58231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.774498] LNet: Removed LNI 192.168.204.20@tcp1 [ 908.049600] Key type .llcrypt unregistered [ 908.051347] Key type ._llcrypt unregistered [ 919.043318] Key type ._llcrypt registered [ 919.044732] Key type .llcrypt registered [ 920.141235] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:19:45 (1763507985) [ 920.867456] Key type .llcrypt unregistered [ 920.869209] Key type ._llcrypt unregistered [ 925.564561] Key type ._llcrypt registered [ 925.566093] Key type .llcrypt registered [ 925.780385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.795669] alg: No test for adler32 (adler32-zlib) [ 926.729863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 926.764313] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 932.320613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 933.060341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e remote_dropped -r 1 [ 933.138971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e remote_dropped -r 1 [ 933.198691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 933.215209] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 933.417324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 933.443557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 933.471231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 933.521910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 933.597353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 933.639325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 933.654241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 933.669895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 933.721812] LNet: 59475:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 933.735888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 933.784366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 933.823979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 933.869452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 933.910513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 933.937760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 933.968153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 934.515087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e remote_error -r 1 [ 934.562714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e remote_error -r 1 [ 934.598596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 934.616160] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 934.785724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 934.807608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 934.829968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 934.855974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 934.876862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 934.893823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 934.955663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 935.015593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 935.078924] LNet: 59584:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 935.100146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 935.141523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 935.177064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 935.221711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 935.254471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 935.270034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 935.301969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 935.703338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e remote_timeout -r 1 [ 935.768815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e remote_timeout -r 1 [ 935.827500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 935.867872] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 936.093519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 936.122080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 936.171840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 936.219696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 936.261897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 936.324309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 936.345104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 936.374610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 936.415194] LNet: 59693:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 936.458661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 936.486869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 936.503859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 936.539773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 936.582426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 936.614453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 936.678180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 936.902798] LNet: 59733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.908484] LNetError: 59733:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.926443] LNet: Removed LNI 192.168.204.20@tcp [ 937.318880] Key type .llcrypt unregistered [ 937.320491] Key type ._llcrypt unregistered [ 951.261584] Key type ._llcrypt registered [ 951.263226] Key type .llcrypt registered [ 952.542919] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:20:18 (1763508018) [ 953.334700] Key type .llcrypt unregistered [ 953.340313] Key type ._llcrypt unregistered [ 957.865231] Key type ._llcrypt registered [ 957.878172] Key type .llcrypt registered [ 958.062980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 958.096676] alg: No test for adler32 (adler32-zlib) [ 959.013610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 959.049134] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 965.003263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 972.077056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 972.103756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 972.115071] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 973.134268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e remote_dropped -r 1 [ 973.173829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e remote_dropped -r 1 [ 973.220928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e remote_dropped -r 1 [ 973.288048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e remote_dropped -r 1 [ 973.349096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e remote_dropped -r 1 [ 973.393364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e remote_dropped -r 1 [ 973.477695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e remote_dropped -r 1 [ 973.534707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e remote_dropped -r 1 [ 973.562481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 973.580454] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 973.595804] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.120@tcp [ 973.856371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 973.901145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 973.944637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 974.003230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 974.090894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 974.129411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 974.187311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 974.226089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 974.262693] LNet: 61194:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 974.285709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 974.357753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 974.410631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 974.446842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 974.475864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 974.506312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 974.549965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 974.581352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 975.344263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e remote_error -r 1 [ 975.418324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e remote_error -r 1 [ 975.460822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e remote_error -r 1 [ 975.490400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e remote_error -r 1 [ 975.516876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e remote_error -r 1 [ 975.560444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e remote_error -r 1 [ 975.619641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e remote_error -r 1 [ 975.682319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e remote_error -r 1 [ 975.754132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 975.799506] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 975.807350] LNet: Skipped 2 previous similar messages [ 976.066024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 976.092904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 976.117520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 976.198332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 976.239248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 976.262574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 976.287691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 976.320215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 976.346508] LNet: 61318:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 976.365795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 976.398361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 976.427198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 976.466157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 976.494238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 976.548033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 976.569943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 976.603113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 977.011517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e remote_timeout -r 1 [ 977.031323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e remote_timeout -r 1 [ 977.069848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e remote_timeout -r 1 [ 977.096484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e remote_timeout -r 1 [ 977.118805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e remote_timeout -r 1 [ 977.143780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e remote_timeout -r 1 [ 977.196187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e remote_timeout -r 1 [ 977.242841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e remote_timeout -r 1 [ 977.276792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.120@tcp [ 977.299841] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 977.533886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 977.562386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 977.592430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 977.613092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 977.641215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 977.675529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 977.727514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 977.769672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 977.800982] LNet: 61441:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 977.854924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 977.906188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 977.951910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 977.998317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 978.040464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 978.083229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 978.142257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 978.214398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 983.421915] LNet: 61518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.430674] LNetError: 61518:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.445867] LNet: Removed LNI 192.168.204.20@tcp [ 983.474709] LNet: Removed LNI 192.168.204.20@tcp1 [ 983.856550] Key type .llcrypt unregistered [ 983.858397] Key type ._llcrypt unregistered [ 998.115943] Key type ._llcrypt registered [ 998.117298] Key type .llcrypt registered [ 999.163856] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:21:04 (1763508064) [ 999.536459] Key type .llcrypt unregistered [ 999.538325] Key type ._llcrypt unregistered [ 1003.735912] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1003.887891] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1003.897107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1003.898784] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1006.358934] Key type ._llcrypt registered [ 1006.360568] Key type .llcrypt registered [ 1006.507165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1006.524727] alg: No test for adler32 (adler32-zlib) [ 1007.442526] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1007.477840] LNet: 62600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.483598] LNetError: 62600:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.492334] LNet: Removed LNI 192.168.204.20@tcp [ 1007.981381] Key type .llcrypt unregistered [ 1007.982900] Key type ._llcrypt unregistered [ 1011.808506] Key type ._llcrypt registered [ 1011.809687] Key type .llcrypt registered [ 1011.972926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.989788] alg: No test for adler32 (adler32-zlib) [ 1012.911944] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1012.919898] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1012.951233] LNet: 62869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.957851] LNetError: 62869:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.965394] LNet: Removed LNI 192.168.204.20@tcp [ 1012.977856] LNet: Removed LNI 10.1.2.3@tcp1 [ 1013.521470] Key type .llcrypt unregistered [ 1013.523072] Key type ._llcrypt unregistered [ 1017.048959] Key type ._llcrypt registered [ 1017.052197] Key type .llcrypt registered [ 1017.217617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1017.233493] alg: No test for adler32 (adler32-zlib) [ 1018.110065] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1018.112035] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1018.153223] LNet: 63138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.158990] LNetError: 63138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.162452] LNet: Removed LNI 192.168.204.20@tcp [ 1018.186471] LNet: Removed LNI 10.1.2.3@tcp [ 1018.624463] Key type .llcrypt unregistered [ 1018.625782] Key type ._llcrypt unregistered [ 1022.562266] Key type ._llcrypt registered [ 1022.564327] Key type .llcrypt registered [ 1022.738556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1022.751170] alg: No test for adler32 (adler32-zlib) [ 1023.672592] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1023.678206] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1023.715824] LNet: 63408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.721988] LNetError: 63408:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.725990] LNet: Removed LNI 192.168.204.20@tcp [ 1023.736406] LNet: Removed LNI 10.1.2.3@tcp [ 1024.322463] Key type .llcrypt unregistered [ 1024.324237] Key type ._llcrypt unregistered [ 1028.213600] Key type ._llcrypt registered [ 1028.215261] Key type .llcrypt registered [ 1028.361139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1028.377152] alg: No test for adler32 (adler32-zlib) [ 1029.294236] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1029.296678] LNetError: 63668:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1029.302980] LNetError: Error -100 starting up LNI tcp [ 1029.310580] LNet: Removed LNI 192.168.204.20@tcp [ 1029.863534] Key type .llcrypt unregistered [ 1029.864670] Key type ._llcrypt unregistered [ 1038.126980] Key type ._llcrypt registered [ 1038.136731] Key type .llcrypt registered [ 1039.238426] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:21:44 (1763508104) [ 1040.054277] Key type .llcrypt unregistered [ 1040.055784] Key type ._llcrypt unregistered [ 1044.371737] Key type ._llcrypt registered [ 1044.373552] Key type .llcrypt registered [ 1044.575406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1044.598873] alg: No test for adler32 (adler32-zlib) [ 1045.501444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1045.519420] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1052.015216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 1052.825205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e network_timeout -r 1 [ 1052.863682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e network_timeout -r 1 [ 1052.897070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 1052.932242] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 1053.212641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1053.241223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1053.261836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1053.301114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1053.337957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1053.358563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1053.386440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1053.422421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.439877] LNet: 64917:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1053.453795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.474696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.496852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.537290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.569953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.630348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.986720] LNet: 64955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.020202] LNetError: 64955:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.048616] LNet: Removed LNI 192.168.204.20@tcp [ 1054.512790] Key type .llcrypt unregistered [ 1054.514500] Key type ._llcrypt unregistered [ 1069.366093] Key type ._llcrypt registered [ 1069.367456] Key type .llcrypt registered [ 1069.536756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1069.549624] alg: No test for adler32 (adler32-zlib) [ 1070.459825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1070.489114] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1076.479449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 1083.193663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.244639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1083.275246] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1084.580563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e network_timeout -r 1 [ 1084.602643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e network_timeout -r 1 [ 1084.627849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e network_timeout -r 1 [ 1084.653629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e network_timeout -r 1 [ 1084.685339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e network_timeout -r 1 [ 1084.721802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e network_timeout -r 1 [ 1084.752248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e network_timeout -r 1 [ 1084.788187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e network_timeout -r 1 [ 1084.810972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 1084.828354] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 1084.833418] LNet: 1 local NIs in recovery (showing 1): 192.168.204.20@tcp [ 1084.835494] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.120@tcp [ 1084.999558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.037232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1085.062186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.093190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.126724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.166642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.196989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1085.226518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1085.264855] LNet: 65828:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1085.275862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1085.302495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1085.329640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1085.353167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1085.387949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1085.434827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.965348] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 1089.261259] LNet: 65901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.269954] LNetError: 65901:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.280731] LNet: Removed LNI 192.168.204.20@tcp [ 1089.293447] LNet: Removed LNI 192.168.204.20@tcp1 [ 1089.606298] Key type .llcrypt unregistered [ 1089.607837] Key type ._llcrypt unregistered [ 1102.724761] Key type ._llcrypt registered [ 1102.726340] Key type .llcrypt registered [ 1103.727478] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:22:49 (1763508169) [ 1104.034255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.041407] alg: No test for adler32 (adler32-zlib) [ 1104.940763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.980531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1105.005067] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1105.569965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1105.587610] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1106.266922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1106.308138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1106.339673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1106.373958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1106.408418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1106.432176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1106.456825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.20@tcp [ 1106.473692] LNetError: 66998:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.20@tcp to recovery [ 1107.045614] LNet: 1 local NIs in recovery (showing 1): 192.168.204.20@tcp [ 1107.047060] LNet: There was an unexpected network error while writing to 192.168.204.20: rc = -22 [ 1109.088413] LNet: There was an unexpected network error while writing to 192.168.204.20: rc = -22 [ 1109.817807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1113.185963] LNet: There was an unexpected network error while writing to 192.168.204.20: rc = -22 [ 1114.382926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1114.413276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1114.463625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1114.484946] LNet: 67115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.491661] LNetError: 67115:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.496303] LNet: Removed LNI 192.168.204.20@tcp [ 1114.510183] LNet: Removed LNI 192.168.204.20@tcp1 [ 1114.535455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1114.591647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1114.633568] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1115.254640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1115.279940] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1115.939807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1115.962857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.007088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1116.031393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1116.050324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1116.077466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1116.100593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1116.123647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.20@tcp [ 1116.135750] LNetError: 67388:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.20@tcp to recovery [ 1116.641457] LNet: There was an unexpected network error while writing to 192.168.204.20: rc = -22 [ 1119.478935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1122.784821] LNet: There was an unexpected network error while writing to 192.168.204.20: rc = -22 [ 1122.789730] LNet: Skipped 1 previous similar message [ 1127.237717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1127.269777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1127.313074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.041278] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:23:18 (1763508198) [ 1132.118716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1132.140683] LNet: 68100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.146473] LNetError: 68100:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1132.160210] LNetError: 68100:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1132.164266] LNet: Removed LNI 192.168.204.20@tcp [ 1132.190019] LNet: Removed LNI 192.168.204.20@tcp1 [ 1132.203545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1132.250503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1132.268195] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1132.829993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1132.848989] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1133.478868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1133.542407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1133.572856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1133.611164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1133.648917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1133.677195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1133.701941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.20@tcp --health 0 [ 1133.776634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1134.304264] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.20@tcp [ 1134.304308] LNet: There was an unexpected network error while writing to 192.168.204.20: rc = -22 [ 1134.310027] LNet: Skipped 1 previous similar message [ 1148.694360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1148.810108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1148.835934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1148.869629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.20@tcp --health 500 [ 1148.950881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1155.494758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1155.544972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1155.559325] LNet: 68596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.564470] LNetError: 68596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.570357] LNetError: 68596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1155.575814] LNet: Removed LNI 192.168.204.20@tcp [ 1155.598695] LNet: Removed LNI 192.168.204.20@tcp1 [ 1155.616123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.658258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1155.683870] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1156.257814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1156.288986] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1156.993347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1157.021422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.078656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1157.117062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1157.161773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1157.195599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1157.228919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1157.254515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.20@tcp --health 0 [ 1158.688395] LNet: There was an unexpected network error while writing to 192.168.204.20: rc = -22 [ 1158.692694] LNet: Skipped 3 previous similar messages [ 1159.574692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1169.526048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1169.551948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.573884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.488491] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:24:00 (1763508240) [ 1174.789517] LNet: 69653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.797464] LNetError: 69653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1174.801356] LNetError: 69653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1174.808433] LNet: Removed LNI 192.168.204.20@tcp [ 1174.819765] LNet: Removed LNI 192.168.204.20@tcp1 [ 1175.327096] Key type .llcrypt unregistered [ 1175.328742] Key type ._llcrypt unregistered [ 1179.557622] Key type ._llcrypt registered [ 1179.561952] Key type .llcrypt registered [ 1179.753385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1179.768568] alg: No test for adler32 (adler32-zlib) [ 1180.684125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1180.715764] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1187.509409] Lustre: DEBUG MARKER: Initial discovery [ 1187.537126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.120@tcp [ 1189.326721] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1189.413320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1189.462876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.120@tcp [ 1189.502192] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 1189.528393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1189.563909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.120@tcp -r 1 -m GET -l 3 [ 1189.587835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.120@tcp -r 1 -m GET -e local_error [ 1189.643785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.120@tcp -r 1 -m PUT -l 6 [ 1189.924634] LNet: 1 local NIs in recovery (showing 1): 192.168.204.20@tcp [ 1190.720280] Lustre: DEBUG MARKER: Force oleg420-client.virtnet to discover 192.168.204.120@tcp (in background) [ 1190.845218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.120@tcp [ 1192.220873] Lustre: DEBUG MARKER: Force oleg420-server to discover 192.168.204.20@tcp [ 1193.955067] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 1194.261392] Lustre: DEBUG MARKER: Wait for 70410 [ 1195.506231] Lustre: DEBUG MARKER: Finished wait on 70410 [ 1195.561067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1195.598735] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 1195.605471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1195.883693] LNet: 70860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.893574] LNetError: 70860:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.968486] LNet: Removed LNI 192.168.204.20@tcp [ 1197.294572] Key type .llcrypt unregistered [ 1197.296056] Key type ._llcrypt unregistered [ 1209.645940] Key type ._llcrypt registered [ 1209.648372] Key type .llcrypt registered [ 1210.869860] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:24:36 (1763508276) [ 1211.385572] Key type .llcrypt unregistered [ 1211.388810] Key type ._llcrypt unregistered [ 1216.491348] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1216.669742] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1216.672171] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1216.695259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1219.049606] Key type ._llcrypt registered [ 1219.050917] Key type .llcrypt registered [ 1219.171819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1219.182342] alg: No test for adler32 (adler32-zlib) [ 1220.043904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1220.095440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1220.114913] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1220.692682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1220.711684] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1227.642787] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:24:53 (1763508293) [ 1227.765326] LNet: 72925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.769842] LNetError: 72925:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.791780] LNet: Removed LNI 192.168.204.20@tcp [ 1227.804782] LNet: Removed LNI 10.1.2.3@tcp [ 1228.404373] Key type .llcrypt unregistered [ 1228.406647] Key type ._llcrypt unregistered [ 1234.041564] Key type ._llcrypt registered [ 1234.043387] Key type .llcrypt registered [ 1234.255288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1234.266762] alg: No test for adler32 (adler32-zlib) [ 1235.217801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1235.273212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1235.292213] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1236.001119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1236.014336] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1237.035388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.20@tcp [ 1238.112285] LNetError: Unexpected error -22 connecting to 192.168.204.20@tcp at host 192.168.204.20:988 [ 1242.283513] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:25:08 (1763508308) [ 1242.344572] LNet: 74049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.357439] LNetError: 74049:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1242.372184] LNet: Removed LNI 192.168.204.20@tcp [ 1243.429937] LNet: Removed LNI 10.1.2.3@tcp [ 1244.045486] Key type .llcrypt unregistered [ 1244.049983] Key type ._llcrypt unregistered [ 1248.683171] Key type ._llcrypt registered [ 1248.684700] Key type .llcrypt registered [ 1248.874193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1248.888673] alg: No test for adler32 (adler32-zlib) [ 1249.806643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.901099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1249.939122] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1250.673975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1250.685279] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1251.478536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.20@tcp1 --nid 192.168.204.20@tcp2 [ 1255.242996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.20@tcp1 192.168.204.20@tcp2 [ 1255.261929] LNetError: 74889:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.20@tcp1 to 192.168.204.20@tcp2 - Route table may be misconfigured [ 1255.278635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.20@tcp2 192.168.204.20@tcp1 [ 1260.217058] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 18:25:26 (1763508326) [ 1260.433727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1260.467545] LNet: 75462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.475524] LNetError: 75462:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1260.481100] LNet: Removed LNI 192.168.204.20@tcp1 [ 1261.543294] LNet: Removed LNI 192.168.204.20@tcp2 [ 1261.578862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1261.629578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1261.672515] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1262.550565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1262.565277] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1263.591976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1263.668989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.20@tcp 192.168.204.20@tcp [ 1263.736750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.20@tcp1 192.168.204.20@tcp1 [ 1263.792886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.20@tcp -d 192.168.204.20@tcp -e network_timeout [ 1263.872590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -e network_timeout [ 1263.927881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -e network_timeout [ 1263.970823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -e network_timeout [ 1264.011685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1264.032104] LNet: There was an unexpected network error while writing to 192.168.204.20: rc = -22 [ 1264.061330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1264.183844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1264.317674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1264.676403] LNet: 1 local NIs in recovery (showing 1): 192.168.204.20@tcp [ 1270.828751] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 18:25:36 (1763508336) [ 1270.992622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1271.044454] LNet: 76324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1271.056366] LNetError: 76324:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1271.061715] LNetError: 76324:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1271.067115] LNet: Removed LNI 192.168.204.20@tcp [ 1272.107845] LNet: Removed LNI 192.168.204.20@tcp1 [ 1272.156670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.235095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1272.478679] LNet: 76361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.887913] Key type .llcrypt unregistered [ 1272.889699] Key type ._llcrypt unregistered [ 1282.918502] Key type ._llcrypt registered [ 1282.920776] Key type .llcrypt registered [ 1284.067931] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 18:25:49 (1763508349) [ 1284.497533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1284.509331] alg: No test for adler32 (adler32-zlib) [ 1285.480444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1286.648819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1288.058226] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 18:25:53 (1763508353) [ 1288.172642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1288.192442] LNet: 77454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.226382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1288.260090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1288.274581] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1288.816698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1288.832580] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1289.560278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1289.586775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp1 [ 1289.619735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp1 [ 1294.841333] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 18:26:00 (1763508360) [ 1296.356247] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg420-server [ 1297.826525] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 18:26:03 (1763508363) [ 1299.280242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg420-server [ 1300.633026] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 18:26:06 (1763508366) [ 1302.282454] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg420-server [ 1303.778088] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 18:26:09 (1763508369) [ 1305.125872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg420-server [ 1306.396842] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 18:26:12 (1763508372) [ 1307.848568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg420-server [ 1309.398492] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 18:26:15 (1763508375) [ 1311.129775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg420-server [ 1312.687732] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 18:26:18 (1763508378) [ 1314.260681] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg420-server [ 1316.025663] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 18:26:21 (1763508381) [ 1317.333483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg420-server [ 1318.682072] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 18:26:24 (1763508384) [ 1320.347064] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg420-server [ 1321.914516] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 18:26:27 (1763508387) [ 1322.020344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1322.081807] LNet: 81326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.089409] LNetError: 81326:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.092430] LNet: Removed LNI 192.168.204.20@tcp [ 1322.112970] LNet: Removed LNI 192.168.204.20@tcp1 [ 1322.155369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1322.206760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1322.229083] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1322.941945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1323.014946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1323.190842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1323.226511] LNet: 81479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.237695] LNetError: 81479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.241126] LNetError: 81479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1324.266888] LNet: Removed LNI 192.168.204.20@tcp [ 1324.297542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.362613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1324.399219] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1325.172686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1325.345262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1325.376994] LNet: 81625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.382725] LNetError: 81625:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.402851] LNet: Removed LNI 192.168.204.20@tcp [ 1325.419300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.478585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1325.509951] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1326.335708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1326.435739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1326.640410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1327.720789] LNet: Removed LNI 192.168.204.20@tcp [ 1327.741304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1327.802953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1327.833214] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1328.459105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1328.698164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1328.723554] LNetError: 81917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.728558] LNetError: 81917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1328.754558] LNet: Removed LNI 192.168.204.20@tcp [ 1328.775361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.835760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1328.868052] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1329.481818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1329.544795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1331.857186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1331.880378] LNet: 82083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.887307] LNet: 82083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1332.907778] LNet: Removed LNI 192.168.204.20@tcp [ 1332.923138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.976531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1332.997830] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1333.568779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1333.775203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1333.793993] LNetError: 82228:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.806141] LNetError: 82228:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1333.817716] LNet: Removed LNI 192.168.204.20@tcp [ 1333.832240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.874846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1333.891671] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1334.416420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1334.502399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1336.897914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1337.960946] LNet: Removed LNI 192.168.204.20@tcp [ 1337.988177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1338.017153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1338.046573] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1338.653074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1338.835833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1339.879216] LNet: Removed LNI 192.168.204.20@tcp [ 1339.919050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.979796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1339.996924] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1340.518809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1340.568664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1340.729265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1340.751832] LNet: 82687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.760746] LNet: 82687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1341.801507] LNet: Removed LNI 192.168.204.20@tcp [ 1341.818847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1341.863862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1341.886854] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1342.431992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1342.614102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1342.623782] LNetError: 82832:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1342.628743] LNetError: 82832:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1342.637801] LNet: Removed LNI 192.168.204.20@tcp [ 1342.651907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1342.688221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1342.713220] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1343.219327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1343.286481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1344.556614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1345.644506] LNet: Removed LNI 192.168.204.20@tcp [ 1345.672746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.728232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1345.751423] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1346.429444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1346.693679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1347.752836] LNet: Removed LNI 192.168.204.20@tcp [ 1347.767650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1347.816759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1347.837273] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1348.535254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1348.609369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1348.659403] LNet: Removed LNI 192.168.204.20@tcp [ 1348.676043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1348.758905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1348.798727] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1349.505415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1355.950691] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 18:27:01 (1763508421) [ 1356.118212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1356.163585] LNet: Removed LNI 192.168.204.20@tcp [ 1356.208081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.280389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1356.305988] LNet: Added LNI 192.168.204.20@tcp231 [8/256/0/180] [ 1357.398196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.416469] LNet: 84110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.419175] LNet: 84110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1357.435676] LNet: Removed LNI 192.168.204.20@tcp231 [ 1357.453914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.552667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1357.590649] LNet: Added LNI 192.168.204.20@tcp231 [8/256/0/180] [ 1357.597252] LNet: 84118:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1357.959090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1358.002870] LNet: Removed LNI 192.168.204.20@tcp231 [ 1358.039930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1358.087550] LNet: Added LNI 192.168.204.20@tcp231 [8/256/0/180] [ 1359.518850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1359.543652] LNetError: 84262:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1359.549896] LNetError: 84262:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1359.566905] LNet: Removed LNI 192.168.204.20@tcp231 [ 1359.590983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1359.699821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1359.717079] LNet: Added LNI 192.168.204.20@tcp231 [8/256/0/0] [ 1359.721613] LNet: 84270:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1360.042983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1360.076682] LNet: Removed LNI 192.168.204.20@tcp231 [ 1360.089384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1360.116939] LNet: Added LNI 192.168.204.20@tcp231 [8/256/0/0] [ 1361.463432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1361.498673] LNet: Removed LNI 192.168.204.20@tcp231 [ 1361.529777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1361.668851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1361.691777] LNet: Added LNI 192.168.204.20@tcp231 [8/256/0/60] [ 1361.697415] LNet: 84421:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1362.005431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1362.050377] LNet: Removed LNI 192.168.204.20@tcp231 [ 1362.064623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1362.104493] LNet: Added LNI 192.168.204.20@tcp231 [8/256/0/60] [ 1363.055808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1363.092713] LNet: Removed LNI 192.168.204.20@tcp231 [ 1363.106273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.213913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1363.248572] LNet: Added LNI 192.168.204.20@tcp231 [8/256/0/180] [ 1363.790518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1363.856233] LNet: Removed LNI 192.168.204.20@tcp231 [ 1363.876887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1363.916708] LNet: Added LNI 192.168.204.20@tcp231 [8/256/0/180] [ 1370.584167] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 18:27:16 (1763508436) [ 1370.671926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.756173] LNet: Removed LNI 192.168.204.20@tcp231 [ 1370.808475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.868598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1370.899777] LNet: Added LNI 192.168.204.20@tcp232 [8/256/0/180] [ 1371.485293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1371.533390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1371.562185] LNet: Removed LNI 192.168.204.20@tcp232 [ 1371.581600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1371.614802] LNet: Added LNI 192.168.204.20@tcp232 [8/256/0/180] [ 1372.184638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1372.216171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1372.242947] LNet: Removed LNI 192.168.204.20@tcp232 [ 1372.258931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1372.284814] LNet: Added LNI 192.168.204.20@tcp232 [8/256/0/180] [ 1372.886366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1372.914459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1372.942470] LNet: Removed LNI 192.168.204.20@tcp232 [ 1372.955710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1372.976397] LNet: Added LNI 192.168.204.20@tcp232 [8/256/0/180] [ 1373.607890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1373.645420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1373.676771] LNet: Removed LNI 192.168.204.20@tcp232 [ 1373.703458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1378.375602] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 18:27:24 (1763508444) [ 1379.260059] Key type .llcrypt unregistered [ 1379.265940] Key type ._llcrypt unregistered [ 1383.599153] Key type ._llcrypt registered [ 1383.600734] Key type .llcrypt registered [ 1383.820836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1383.842459] alg: No test for adler32 (adler32-zlib) [ 1384.731723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1384.749515] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1389.712902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 1395.210145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.236702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1395.255595] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1396.246811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_interrupt -r 2 [ 1396.268697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_interrupt -r 2 [ 1396.294241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_interrupt -r 2 [ 1396.318283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_interrupt -r 2 [ 1396.343528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_interrupt -r 2 [ 1396.374531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_interrupt -r 2 [ 1396.399725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_interrupt -r 2 [ 1396.424817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_interrupt -r 2 [ 1396.456213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.20@tcp 192.168.204.120@tcp [ 1396.469106] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 1396.621837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1396.653348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1396.671449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1396.694867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1396.722417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1396.740299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1396.765673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1396.797037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1396.820880] LNet: 87070:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1396.842375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1396.877485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1396.898469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1396.925759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1396.950973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1396.979498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1397.400761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_dropped -r 2 [ 1397.444755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_dropped -r 2 [ 1397.471850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_dropped -r 2 [ 1397.508179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_dropped -r 2 [ 1397.551472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_dropped -r 2 [ 1397.588546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_dropped -r 2 [ 1397.617791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_dropped -r 2 [ 1397.637709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_dropped -r 2 [ 1397.664989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.20@tcp 192.168.204.120@tcp [ 1397.680285] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 1397.684781] LNet: 1 local NIs in recovery (showing 1): 192.168.204.20@tcp [ 1397.882184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1397.904447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1397.928373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1397.962805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1397.987231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1398.012358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1398.038137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1398.069437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1398.089981] LNet: 87190:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1398.111539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1398.134274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1398.160917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1398.191543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1398.211840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1398.235810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1398.689687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_aborted -r 2 [ 1398.725001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_aborted -r 2 [ 1398.766367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_aborted -r 2 [ 1398.790364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_aborted -r 2 [ 1398.822732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_aborted -r 2 [ 1398.855440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_aborted -r 2 [ 1398.898676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_aborted -r 2 [ 1398.936087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_aborted -r 2 [ 1398.966535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.20@tcp 192.168.204.120@tcp [ 1398.993836] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 1399.200721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1399.225539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1399.246896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1399.271048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1399.301915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1399.325928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1399.354370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1399.376505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1399.394628] LNet: 87310:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1399.405471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1399.428588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1399.449748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1399.475089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1399.507172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1399.532983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1399.917117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_no_route -r 2 [ 1399.938877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_no_route -r 2 [ 1399.959906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_no_route -r 2 [ 1399.983691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_no_route -r 2 [ 1400.010228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_no_route -r 2 [ 1400.036387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_no_route -r 2 [ 1400.069334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_no_route -r 2 [ 1400.099409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_no_route -r 2 [ 1400.128912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.20@tcp 192.168.204.120@tcp [ 1400.285682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1400.315029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1400.342041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1400.364290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1400.387808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1400.414628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1400.439637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1400.465032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1400.501912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1400.539468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1400.585243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1400.609518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1400.631865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1400.664803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1401.027400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e local_timeout -r 2 [ 1401.047543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e local_timeout -r 2 [ 1401.071259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e local_timeout -r 2 [ 1401.098118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e local_timeout -r 2 [ 1401.129599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e local_timeout -r 2 [ 1401.165720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e local_timeout -r 2 [ 1401.192367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e local_timeout -r 2 [ 1401.223384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e local_timeout -r 2 [ 1401.266771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.20@tcp 192.168.204.120@tcp [ 1401.283302] LNet: There was an unexpected network error while writing to 192.168.204.120: rc = -22 [ 1401.286715] LNet: Skipped 1 previous similar message [ 1401.523080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1401.543790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1401.567953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1401.610740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1401.642246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1401.690143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1401.727945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1401.756588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1401.783489] LNet: 87551:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1401.786902] LNet: 87551:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1401.805515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1401.834557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1401.862796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.898308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1401.935937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1401.980520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1402.463629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp -m GET -e remote_dropped -r 2 [ 1402.524598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.120@tcp1 -m GET -e remote_dropped -r 2 [ 1402.572235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp -m GET -e remote_dropped -r 2 [ 1402.619971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp -d 192.168.204.20@tcp1 -m GET -e remote_dropped -r 2 [ 1402.652800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp -m GET -e remote_dropped -r 2 [ 1402.709809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.120@tcp1 -m GET -e remote_dropped -r 2 [ 1402.740828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp -m GET -e remote_dropped -r 2 [ 1402.778579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.20@tcp1 -d 192.168.204.20@tcp1 -m GET -e remote_dropped -r 2 [ 1402.828121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.20@tcp 192.168.204.120@tcp [ 1402.856839] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.120@tcp [ 1403.078361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1403.106705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1403.127928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1403.156423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1403.180333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1403.218517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1403.240636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1403.270253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1403.296594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1403.324396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1403.352119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1403.391306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1403.431182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1403.465367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1406.793711] LNet: 87744:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.799440] LNetError: 87744:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.804086] LNet: Removed LNI 192.168.204.20@tcp [ 1406.811920] LNet: Removed LNI 192.168.204.20@tcp1 [ 1407.064589] Key type .llcrypt unregistered [ 1407.066266] Key type ._llcrypt unregistered [ 1416.889426] Key type ._llcrypt registered [ 1416.890955] Key type .llcrypt registered [ 1417.825697] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 18:28:03 (1763508483) [ 1418.092277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1418.101303] alg: No test for adler32 (adler32-zlib) [ 1418.969871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.011736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.035156] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1419.629300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1419.650617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.20@tcp [ 1419.662903] LNetError: 88718:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.20@tcp to recovery [ 1419.742830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.20@tcp [ 1419.807070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1424.621203] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 18:28:10 (1763508490) [ 1424.718961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1424.743459] LNet: 89301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1424.747963] LNetError: 89301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.766307] LNet: Removed LNI 192.168.204.20@tcp [ 1424.792864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.863560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1424.891266] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1425.532376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1425.547168] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1426.412856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.20@tcp [ 1426.434626] LNetError: 89574:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.20@tcp to recovery [ 1426.912794] LNet: 1 local NIs in recovery (showing 1): 192.168.204.20@tcp [ 1426.914271] LNet: There was an unexpected network error while writing to 192.168.204.20: rc = -22 [ 1427.645568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.20@tcp [ 1434.094950] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 18:28:20 (1763508500) [ 1435.197701] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg420-server [ 1436.219157] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 18:28:22 (1763508502) [ 1436.306746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1436.330980] LNet: 90528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.337513] LNetError: 90528:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.344296] LNet: Removed LNI 192.168.204.20@tcp [ 1437.346138] LNetError: 90528:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.360683] LNet: Removed LNI 192.168.204.20@tcp1 [ 1437.386464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.449543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1437.472463] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1437.957149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1438.012078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1438.038970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1438.093021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1438.123757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1438.157564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1442.489466] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 18:28:28 (1763508508) [ 1442.586595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1442.612302] LNet: 91254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1442.617833] LNetError: 91254:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.636166] LNet: Removed LNI 192.168.204.20@tcp [ 1442.650539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.694498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1442.724445] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1449.041538] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 18:28:34 (1763508514) [ 1449.200426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.239699] LNet: 91955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.252175] LNetError: 91955:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.280168] LNet: Removed LNI 192.168.204.20@tcp [ 1449.306221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.358895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1449.387659] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1455.547321] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 18:28:41 (1763508521) [ 1457.135325] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1458.787516] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 18:28:44 (1763508524) [ 1459.351184] LNet: 92992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.362962] LNetError: 92992:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.382124] LNet: Removed LNI 192.168.204.20@tcp [ 1460.020803] Key type .llcrypt unregistered [ 1460.022982] Key type ._llcrypt unregistered [ 1464.606225] Key type ._llcrypt registered [ 1464.620230] Key type .llcrypt registered [ 1464.775576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1464.786896] alg: No test for adler32 (adler32-zlib) [ 1465.665341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1465.706486] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1471.298947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 1474.662219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.120@tcp [ 1475.066417] LNet: 93404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1475.073439] LNetError: 93404:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1475.088876] LNet: Removed LNI 192.168.204.20@tcp [ 1475.451387] Key type .llcrypt unregistered [ 1475.455168] Key type ._llcrypt unregistered [ 1484.044255] Key type ._llcrypt registered [ 1484.045469] Key type .llcrypt registered [ 1485.054267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1486.261394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1487.506871] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 18:29:13 (1763508553) [ 1487.825455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1487.838331] alg: No test for adler32 (adler32-zlib) [ 1488.718591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.770169] LNet: 94488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.204430] Key type .llcrypt unregistered [ 1489.206299] Key type ._llcrypt unregistered [ 1493.212318] Key type ._llcrypt registered [ 1493.214950] Key type .llcrypt registered [ 1493.330207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1493.347500] alg: No test for adler32 (adler32-zlib) [ 1494.234413] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1494.236650] LNet: 94740:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1494.272815] LNetError: 94752:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.21@tcp is being used as a gateway but routing feature is not turned on [ 1494.283153] LNetError: 94752:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.21@tcp has gone from up to down [ 1494.604989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1494.623902] LNet: 94798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1494.628958] LNetError: 94798:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1494.657694] LNet: Removed LNI 192.168.204.20@tcp [ 1494.704512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.739521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1494.764289] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1494.769525] LNet: 94805:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1494.775236] LNet: 94805:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1494.781734] LNetError: 94805:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.22@tcp is a gateway. Can not delete it [ 1494.786097] LNetError: 94805:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.204.22@tcp is a router. Can not be deleted [ 1494.796564] LNet: 94805:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1494.808464] LNetError: 94802:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.24@tcp is being used as a gateway but routing feature is not turned on [ 1494.819924] LNetError: 94802:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 1494.830888] LNetError: 94802:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.24@tcp has gone from up to down [ 1494.838631] LNetError: 94802:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1499.643609] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 18:29:25 (1763508565) [ 1500.992261] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg420-server [ 1502.424520] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 18:29:28 (1763508568) [ 1504.102051] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg420-server [ 1505.615334] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 18:29:31 (1763508571) [ 1506.804783] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 1508.648169] Key type lgssc registered [ 1509.470440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1522.750619] Key type lgssc unregistered [ 1522.991536] LNet: 97219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.999689] LNetError: 97219:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.007252] LNet: Removed LNI 192.168.204.20@tcp [ 1524.421129] Key type .llcrypt unregistered [ 1524.426035] Key type ._llcrypt unregistered [ 1533.504329] Key type ._llcrypt registered [ 1533.505809] Key type .llcrypt registered [ 1534.341992] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 18:30:00 (1763508600) [ 1534.611889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1534.619544] alg: No test for adler32 (adler32-zlib) [ 1535.471888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.534950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1535.574541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1535.629415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1535.671833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1539.397690] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 18:30:05 (1763508605) [ 1539.453471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.469274] LNet: 98627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.486920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.527770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.545295] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1540.086076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.099398] LNet: 98776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.104808] LNetError: 98776:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.123355] LNet: Removed LNI 192.168.204.20@tcp [ 1540.142846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.185936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1540.206812] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1540.495451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.535071] LNet: Removed LNI 192.168.204.20@tcp [ 1540.553156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.592084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1540.613238] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1544.643664] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 18:30:10 (1763508610) [ 1544.695759] LNet: 99387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.699657] LNet: 99387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1544.708780] LNetError: 99387:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.712046] LNetError: 99387:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1544.730465] LNet: Removed LNI 192.168.204.20@tcp [ 1545.154351] Key type .llcrypt unregistered [ 1545.156299] Key type ._llcrypt unregistered [ 1549.064467] Key type ._llcrypt registered [ 1549.065734] Key type .llcrypt registered [ 1549.245976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1549.263158] alg: No test for adler32 (adler32-zlib) [ 1550.161812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.237611] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1550.239915] LNet: *** cfs_fail_loc=e003, val=100*** [ 1550.757407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1550.771547] LNet: *** cfs_fail_loc=e003, val=100*** [ 1550.773481] LNet: Skipped 2 previous similar messages [ 1550.777631] Ping reply received with mapping: [ 1550.779243] found metadata reply NI[192.168.204.20@tcp] [ 1550.782348] 14 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1550.784414] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1550.786273] 5a 5a 5a 5a 5a 5a 5a 5a [ 1550.886273] LNet: 99789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.890623] LNetError: 99789:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.909830] LNet: Removed LNI 192.168.204.20@tcp [ 1551.330551] Key type .llcrypt unregistered [ 1551.334500] Key type ._llcrypt unregistered [ 1558.894152] Key type ._llcrypt registered [ 1558.898145] Key type .llcrypt registered [ 1559.995485] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 18:30:25 (1763508625) [ 1560.462323] Key type .llcrypt unregistered [ 1560.464927] Key type ._llcrypt unregistered [ 1565.961404] Key type ._llcrypt registered [ 1565.966606] Key type .llcrypt registered [ 1566.151511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1566.167335] alg: No test for adler32 (adler32-zlib) [ 1567.225840] LNetError: 100824:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1567.529503] Key type .llcrypt unregistered [ 1567.532149] Key type ._llcrypt unregistered [ 1571.537095] Key type ._llcrypt registered [ 1571.544454] Key type .llcrypt registered [ 1572.615603] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 18:30:38 (1763508638) [ 1572.962303] Key type .llcrypt unregistered [ 1572.966637] Key type ._llcrypt unregistered [ 1577.370647] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1577.372764] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1578.761638] Key type ._llcrypt registered [ 1578.763262] Key type .llcrypt registered [ 1578.965457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1578.975975] alg: No test for adler32 (adler32-zlib) [ 1579.857183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.946718] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1587.372134] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 18:30:53 (1763508653) [ 1587.665652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.683792] LNet: 102589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.691363] LNetError: 102589:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1587.703568] LNet: Removed LNI 10.1.2.3@tcp [ 1587.726243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.417551] Key type .llcrypt unregistered [ 1588.421383] Key type ._llcrypt unregistered [ 1593.337653] Key type ._llcrypt registered [ 1593.339238] Key type .llcrypt registered [ 1593.507183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1593.528496] alg: No test for adler32 (adler32-zlib) [ 1594.452230] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1594.511948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1594.677215] LNet: 102873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.687544] LNetError: 102873:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.755491] LNet: Removed LNI 192.168.204.20@tcp [ 1596.368519] Key type .llcrypt unregistered [ 1596.370154] Key type ._llcrypt unregistered [ 1604.539682] Key type ._llcrypt registered [ 1604.541686] Key type .llcrypt registered [ 1605.760760] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 18:31:11 (1763508671) [ 1606.457398] Key type .llcrypt unregistered [ 1606.459406] Key type ._llcrypt unregistered [ 1611.555286] Key type ._llcrypt registered [ 1611.560774] Key type .llcrypt registered [ 1611.743043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1611.751884] alg: No test for adler32 (adler32-zlib) [ 1617.705199] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 18:31:23 (1763508683) [ 1617.814168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.845962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.890223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1617.913918] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1618.498816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.543438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1618.575341] LNet: Removed LNI 192.168.204.20@tcp [ 1618.798134] LNet: 104609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1619.182553] Key type .llcrypt unregistered [ 1619.187113] Key type ._llcrypt unregistered [ 1627.278687] Key type ._llcrypt registered [ 1627.281494] Key type .llcrypt registered [ 1628.106540] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 18:31:34 (1763508694) [ 1628.452448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1628.474182] alg: No test for adler32 (adler32-zlib) [ 1629.329435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.368489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1629.392428] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1629.876761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1629.908640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.20@tcp [ 1629.923403] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1629.925893] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1629.929162] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1629, 0, 0 [ 1629.934026] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1629.936685] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1629.940182] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1629.944777] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1629, 0, 0 [ 1629.948193] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1629.952412] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1629.955612] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1629.959638] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1629, 0, 1679 [ 1629.965511] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1629.970193] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1629.974158] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1629.975801] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1629, 0, 1679 [ 1629.981685] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1634.221577] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 18:31:40 (1763508700) [ 1634.480989] LNet: 106144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.486598] LNetError: 106144:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.497935] LNet: Removed LNI 192.168.204.20@tcp [ 1634.901384] Key type .llcrypt unregistered [ 1634.903856] Key type ._llcrypt unregistered [ 1638.380550] Key type ._llcrypt registered [ 1638.382248] Key type .llcrypt registered [ 1638.508884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1638.517071] alg: No test for adler32 (adler32-zlib) [ 1639.365996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1639.383492] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1644.501541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 1650.330621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.371987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1650.391499] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1651.033070] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1651.135749] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1651.142080] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1651.155328] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1653.426606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1653.442137] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1660.428377] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 18:32:06 (1763508726) [ 1660.624032] LNet: 107641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.630400] LNetError: 107641:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.637577] LNet: Removed LNI 192.168.204.20@tcp [ 1661.669213] LNet: Removed LNI 192.168.204.20@tcp1 [ 1661.674110] LNetError: 107641:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1661.689749] LNetError: 107641:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1661.712408] LNet: Removed LNI 10.1.2.3@tcp99 [ 1662.149501] Key type .llcrypt unregistered [ 1662.150832] Key type ._llcrypt unregistered [ 1665.915898] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1665.919441] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1668.311557] Key type ._llcrypt registered [ 1668.314272] Key type .llcrypt registered [ 1668.451101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1668.461488] alg: No test for adler32 (adler32-zlib) [ 1669.364896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.418673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.440503] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1669.983672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1669.996769] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1671.966835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1671.995650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1672.035223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1672.048783] LNetError: 108380:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1672.074432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1672.109908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1672.542731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1672.622931] LNet: 108400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.632076] LNetError: 108400:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.637194] LNet: Removed LNI 192.168.204.20@tcp [ 1672.645987] LNet: Removed LNI 10.1.2.3@tcp [ 1673.227446] Key type .llcrypt unregistered [ 1673.231151] Key type ._llcrypt unregistered [ 1681.985080] Key type ._llcrypt registered [ 1681.986772] Key type .llcrypt registered [ 1682.940036] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 18:32:28 (1763508748) [ 1683.456334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1683.471712] alg: No test for adler32 (adler32-zlib) [ 1684.405902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.485851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.538147] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1685.142881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.20@tcp [ 1685.220725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg420-client@tcp [ 1690.134837] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 18:32:35 (1763508755) [ 1690.292221] LNet: 109930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.302244] LNetError: 109930:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.322353] LNet: Removed LNI 192.168.204.20@tcp [ 1690.818491] Key type .llcrypt unregistered [ 1690.823821] Key type ._llcrypt unregistered [ 1695.386996] Key type ._llcrypt registered [ 1695.391411] Key type .llcrypt registered [ 1695.537437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1695.556128] alg: No test for adler32 (adler32-zlib) [ 1696.412663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1696.459915] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1696.494584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1701.259234] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 18:32:47 (1763508767) [ 1702.354238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1703.586773] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 18:32:49 (1763508769) [ 1704.752852] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1706.325455] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 18:32:51 (1763508771) [ 1706.495331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1706.517917] LNet: 111343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.523533] LNetError: 111343:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.535757] LNet: Removed LNI 192.168.204.20@tcp [ 1706.561459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.619212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1706.644854] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1707.432982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1707.497856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1712.721936] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 18:32:58 (1763508778) [ 1712.849068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.882523] LNet: 112051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.905645] LNetError: 112051:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.960893] LNet: Removed LNI 192.168.204.20@tcp [ 1713.986492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.053050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1714.079572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1714.295747] LNet: 112089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.649393] Key type .llcrypt unregistered [ 1714.651104] Key type ._llcrypt unregistered [ 1723.396640] Key type ._llcrypt registered [ 1723.398198] Key type .llcrypt registered [ 1724.407491] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 18:33:10 (1763508790) [ 1724.862866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1724.884714] alg: No test for adler32 (adler32-zlib) [ 1725.756591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.838223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.859704] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1726.518302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1726.563631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1731.346790] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 18:33:17 (1763508797) [ 1731.449098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.485059] LNet: 113599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.496163] LNetError: 113599:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.515816] LNet: Removed LNI 192.168.204.20@tcp [ 1731.540510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.590138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1731.610250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1736.591352] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 18:33:22 (1763508802) [ 1736.784254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.811291] LNet: 114185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.828987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.873805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1737.239348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1737.262129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1737.326648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1737.681645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1737.717690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1737.792705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1737.819561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.869989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1738.123630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1738.156582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1738.198216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1738.534324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1738.559207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1743.553574] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 18:33:29 (1763508809) [ 1743.611900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.639352] LNet: 114811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.642850] LNet: 114811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1743.661926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.734591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1743.768161] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1744.286830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1744.300085] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1744.883914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1744.928185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1745.070322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.084844] LNetError: 115098:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.088646] LNet: Removed LNI 192.168.204.20@tcp [ 1745.095374] LNet: Removed LNI 192.168.204.20@tcp2 [ 1745.113459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.163756] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1745.169078] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1745.172669] LNet: 115104:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1745.447145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1745.609521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.656167] LNet: Removed LNI 192.168.204.20@tcp [ 1746.658142] LNetError: 115141:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.667969] LNetError: 115141:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1746.696890] LNet: Removed LNI 192.168.204.20@tcp2 [ 1746.716047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.762810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.783412] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1747.383872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1747.400844] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1748.068780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1748.110754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1748.176256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1748.351490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.377579] LNet: 115425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.380813] LNet: 115425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1749.408148] LNet: Removed LNI 192.168.204.20@tcp [ 1749.414403] LNetError: 115425:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.424339] LNetError: 115425:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1749.443366] LNet: Removed LNI 192.168.204.20@tcp2 [ 1749.462513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.509221] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1749.513292] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1749.516223] LNet: 115432:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1749.883498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1750.084050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1750.107391] LNet: Removed LNI 192.168.204.20@tcp [ 1751.146433] LNet: Removed LNI 192.168.204.20@tcp2 [ 1751.164956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.190386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1751.219040] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1751.804188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1751.816587] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1752.521891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1752.572181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1752.800911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.819916] LNet: 115748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.823787] LNet: 115748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1753.888368] LNet: Removed LNI 192.168.204.20@tcp [ 1753.890307] LNetError: 115748:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.897412] LNetError: 115748:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1753.928534] LNet: Removed LNI 192.168.204.20@tcp2 [ 1753.956611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.023844] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1754.034103] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1754.038725] LNet: 115754:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1754.363486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1754.515934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.552663] LNet: Removed LNI 192.168.204.20@tcp [ 1755.569845] LNet: Removed LNI 192.168.204.20@tcp2 [ 1755.609358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.685398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1755.702803] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1756.214394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1756.232395] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1756.946814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1757.004681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1757.064758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1757.243668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.304170] LNet: Removed LNI 192.168.204.20@tcp [ 1759.336957] LNet: Removed LNI 192.168.204.20@tcp2 [ 1759.359468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.404562] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1759.409745] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1759.413352] LNet: 116082:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1759.868165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1760.095900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.115307] LNet: Removed LNI 192.168.204.20@tcp [ 1761.190887] LNet: Removed LNI 192.168.204.20@tcp2 [ 1761.230831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.280322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.299281] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1761.861293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1761.878157] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1762.623763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1762.653929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1762.694335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1762.964913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.998849] LNet: 116415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.003608] LNet: 116415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1763.013138] LNetError: 116415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.016786] LNetError: 116415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1764.064134] LNet: Removed LNI 192.168.204.20@tcp [ 1764.079854] LNet: Removed LNI 192.168.204.20@tcp2 [ 1764.100346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.141165] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1764.149300] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1764.152850] LNet: 116422:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1764.473532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1764.698298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.728138] LNet: Removed LNI 192.168.204.20@tcp [ 1765.748811] LNet: Removed LNI 192.168.204.20@tcp2 [ 1765.792804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.838676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.856693] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1766.482332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1766.501806] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1767.265554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1767.309155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1767.336930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1767.380635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1767.680826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.737246] LNet: Removed LNI 192.168.204.20@tcp [ 1769.772150] LNet: Removed LNI 192.168.204.20@tcp2 [ 1769.820480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.895667] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1769.910105] LNet: Added LNI 192.168.204.20@tcp2 [8/256/0/180] [ 1770.355483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1777.255458] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1778.749417] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 18:34:04 (1763508844) [ 1778.898838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.923549] LNet: Removed LNI 192.168.204.20@tcp [ 1779.943674] LNet: Removed LNI 192.168.204.20@tcp2 [ 1779.966588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.301464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1780.328640] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1781.151600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1781.168063] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1782.151657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.178465] LNet: 117813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.184327] LNet: 117813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1782.194132] LNetError: 117813:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.197633] LNetError: 117813:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1782.217306] LNet: Removed LNI 192.168.204.20@tcp [ 1782.227729] LNet: Removed LNI 10.1.2.3@tcp [ 1782.248346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.318449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1782.354180] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1782.357509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1782.710669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1782.744844] LNet: Removed LNI 192.168.204.20@tcp [ 1782.768899] LNet: Removed LNI 10.1.2.3@tcp [ 1788.347916] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 18:34:14 (1763508854) [ 1788.504303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.564928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.679585] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1788.821487] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1788.825832] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1788.828299] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1791.152573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1791.169175] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1791.819645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1791.835624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1793.639709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.20@tcp [ 1793.667552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1793.716289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.20@tcp -m PUT -e local_timeout -r 1 [ 1793.769822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1793.821810] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1793.826614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1794.034255] LNetError: 118479:0:(lib-move.c:4947:lnet_parse()) 192.168.204.20@tcp, src 192.168.204.20@tcp: Dropping PUT (error -108 looking up sender) [ 1796.064152] LNet: Removed LNI 192.168.204.20@tcp [ 1796.077589] LNet: Removed LNI 10.1.2.3@tcp [ 1796.415265] Key type .llcrypt unregistered [ 1796.417129] Key type ._llcrypt unregistered [ 1804.785513] Key type ._llcrypt registered [ 1804.787091] Key type .llcrypt registered [ 1805.853150] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 18:34:31 (1763508871) [ 1806.216209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1806.229910] alg: No test for adler32 (adler32-zlib) [ 1807.143095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.459605] LNet: 119787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.972347] Key type .llcrypt unregistered [ 1807.973913] Key type ._llcrypt unregistered [ 1812.662566] Key type ._llcrypt registered [ 1812.668303] Key type .llcrypt registered [ 1812.846360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1812.859520] alg: No test for adler32 (adler32-zlib) [ 1813.749549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1813.780286] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1819.227667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 1819.347824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1819.794914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.120@tcp [ 1819.938595] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.120@tcp [ 1820.013796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1820.034143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1820.059242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1820.086887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1820.118236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1820.137347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1820.166184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1820.195585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1820.206653] LNet: 120231:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1820.219145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1820.254547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1820.285478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1820.308990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1820.337872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1820.369941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1820.890482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1821.105562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1821.125855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1821.160417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1821.221560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1821.270862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1821.304239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1821.323679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1821.352285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1821.365422] LNet: 120348:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1821.379553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1821.410769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1821.438228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1821.465954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1821.497658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1821.526922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1822.053994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.120@tcp [ 1822.221701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1822.243956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1822.265243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1822.297558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1822.331991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1822.386648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1822.422475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1822.446313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1822.484833] LNet: 120465:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1822.497219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1822.525282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1822.553311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1822.577615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1822.603374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1822.632140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1823.206143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1823.377305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1823.408848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1823.431036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1823.454732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1823.487873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1823.527774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1823.561806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1823.606166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1823.638757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1823.670571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1823.693917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1823.721602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1823.752478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1823.780165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1824.008475] LNet: 120627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.013806] LNetError: 120627:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.029665] LNet: Removed LNI 192.168.204.20@tcp [ 1824.364460] Key type .llcrypt unregistered [ 1824.366877] Key type ._llcrypt unregistered [ 1836.474145] Key type ._llcrypt registered [ 1836.475569] Key type .llcrypt registered [ 1837.415487] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 18:35:03 (1763508903) [ 1837.674110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1837.689403] alg: No test for adler32 (adler32-zlib) [ 1838.600201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1838.881536] LNet: 121510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.311965] Key type .llcrypt unregistered [ 1839.313727] Key type ._llcrypt unregistered [ 1843.081159] Key type ._llcrypt registered [ 1843.082458] Key type .llcrypt registered [ 1843.233669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1843.248746] alg: No test for adler32 (adler32-zlib) [ 1844.166651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1844.190078] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1849.260708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.120@tcp [ 1855.029221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.053288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1855.080517] LNet: Added LNI 192.168.204.20@tcp1 [8/256/0/180] [ 1855.800152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1856.281974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.120@tcp [ 1856.497731] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.120@tcp [ 1856.530492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1856.556755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1856.594145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1856.626615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1856.657717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1856.705929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1856.731559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1856.769030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1856.785492] LNet: 122157:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1856.799820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1856.835963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1856.880228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1856.924948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1856.970925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1857.006850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1857.483629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1857.676153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1857.719765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1857.760158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1857.793575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1857.829711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1857.884573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1857.912313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1857.939824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1857.971221] LNet: 122274:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1857.988054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1858.032251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1858.075927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1858.098482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1858.127667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1858.156314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1858.664828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.120@tcp [ 1858.851989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1858.871124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1858.896939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1858.924283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1858.961043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1858.993812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1859.043658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1859.076989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1859.091359] LNet: 122391:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1859.109730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1859.138424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1859.164285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1859.207311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1859.254612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1859.278980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1859.898428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1860.049761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1860.070872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1860.104257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1860.128860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1860.158619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1860.198900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1860.229546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1860.266372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1860.295767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1860.343261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1860.391591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1860.427239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1860.468832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1860.509363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1864.135965] LNet: 122588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.142611] LNetError: 122588:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.151759] LNet: Removed LNI 192.168.204.20@tcp [ 1864.164220] LNet: Removed LNI 192.168.204.20@tcp1 [ 1864.531735] Key type .llcrypt unregistered [ 1864.532932] Key type ._llcrypt unregistered [ 1877.529152] Key type ._llcrypt registered [ 1877.530878] Key type .llcrypt registered [ 1878.403408] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1671 sec ========== 18:35:44 (1763508944) [ 1878.872791] Key type .llcrypt unregistered [ 1878.874916] Key type ._llcrypt unregistered [ 1887.609763] Key type ._llcrypt registered [ 1887.611525] Key type .llcrypt registered [ 1887.803807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1887.820866] alg: No test for adler32 (adler32-zlib) [ 1888.852971] Lustre: Lustre: Build Version: 2.16.61_43_g35e72c9 [ 1889.076644] LNet: Added LNI 192.168.204.20@tcp [8/256/0/180] [ 1890.760368] Key type lgssc registered [ 1891.882077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1933.996672] Lustre: Mounted lustre-client [ 1937.487342] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1946.829754] LustreError: 125810:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8b4444892000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1946.858620] LustreError: 125810:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1946.892474] Lustre: Unmounted lustre-client [ 1980.204676] Key type lgssc unregistered [ 1980.410476] LNet: 126292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.418149] LNetError: 126292:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.435305] LNet: Removed LNI 192.168.204.20@tcp [ 1980.834187] Key type .llcrypt unregistered [ 1980.835892] Key type ._llcrypt unregistered