[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 625845727 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.002416] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.008392] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010020] pid_max: default: 32768 minimum: 301 [ 0.011358] LSM: Security Framework initializing [ 0.012080] Yama: becoming mindful. [ 0.013059] SELinux: Initializing. [ 0.014091] *** VALIDATE selinux *** [ 0.026182] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.032488] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.034614] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.036332] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.037172] *** VALIDATE tmpfs *** [ 0.039775] *** VALIDATE proc *** [ 0.041055] *** VALIDATE cgroup *** [ 0.042013] *** VALIDATE cgroup2 *** [ 0.043350] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.044200] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.046008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.048039] Spectre V2 : User space: Vulnerable [ 0.049020] Speculative Store Bypass: Vulnerable [ 0.053120] debug: unmapping init [mem 0xffffffffbb059000-0xffffffffbb060fff] [ 0.056000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.057500] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.058049] ... version: 2 [ 0.059034] ... bit width: 48 [ 0.060024] ... generic registers: 4 [ 0.061034] ... value mask: 0000ffffffffffff [ 0.062034] ... max period: 00007fffffffffff [ 0.063050] ... fixed-purpose events: 3 [ 0.064029] ... event mask: 000000070000000f [ 0.065859] rcu: Hierarchical SRCU implementation. [ 0.068864] smp: Bringing up secondary CPUs ... [ 0.070434] x86: Booting SMP configuration: [ 0.071072] .... node #0, CPUs: #1 #2 #3 [ 0.077093] smp: Brought up 1 node, 4 CPUs [ 0.079017] smpboot: Max logical packages: 1 [ 0.080085] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.159501] node 0 deferred pages initialised in 77ms [ 0.167024] devtmpfs: initialized [ 0.168710] x86/mm: Memory block size: 128MB [ 0.175173] gcov: version magic: 0x41383552 [ 0.181311] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.187228] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.191509] pinctrl core: initialized pinctrl subsystem [ 0.195355] [ 0.196009] ************************************************************* [ 0.201377] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.205016] ** ** [ 0.210015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.214183] ** ** [ 0.219021] ** This means that this kernel is built to expose internal ** [ 0.225066] ** IOMMU data structures, which may compromise security on ** [ 0.228020] ** your system. ** [ 0.232021] ** ** [ 0.235020] ** If you see this message and you are not debugging the ** [ 0.240102] ** kernel, report this immediately to your vendor! ** [ 0.245069] ** ** [ 0.249061] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.253077] ************************************************************* [ 0.260131] NET: Registered protocol family 16 [ 0.264000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.268087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.273095] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.279249] cpuidle: using governor menu [ 0.283123] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.287569] PCI: Using configuration type 1 for base access [ 0.290245] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.308000] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.311266] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.317289] cryptd: max_cpu_qlen set to 1000 [ 0.320028] ACPI: Added _OSI(Module Device) [ 0.323237] ACPI: Added _OSI(Processor Device) [ 0.326025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.329021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.332000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.336000] ACPI: Interpreter enabled [ 0.339464] ACPI: PM: (supports S0 S3 S4 S5) [ 0.343020] ACPI: Using IOAPIC for interrupt routing [ 0.346238] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.354516] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.388338] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.392071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.397043] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.405213] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.414973] acpiphp: Slot [2] registered [ 0.417376] acpiphp: Slot [5] registered [ 0.421221] acpiphp: Slot [6] registered [ 0.423224] acpiphp: Slot [3] registered [ 0.425472] acpiphp: Slot [4] registered [ 0.428114] acpiphp: Slot [7] registered [ 0.431168] acpiphp: Slot [8] registered [ 0.434174] acpiphp: Slot [9] registered [ 0.437260] acpiphp: Slot [10] registered [ 0.441212] acpiphp: Slot [11] registered [ 0.445230] acpiphp: Slot [12] registered [ 0.447255] acpiphp: Slot [13] registered [ 0.449118] acpiphp: Slot [14] registered [ 0.453290] acpiphp: Slot [15] registered [ 0.456221] acpiphp: Slot [16] registered [ 0.459287] acpiphp: Slot [17] registered [ 0.462326] acpiphp: Slot [18] registered [ 0.464232] acpiphp: Slot [19] registered [ 0.466134] acpiphp: Slot [20] registered [ 0.468126] acpiphp: Slot [21] registered [ 0.470120] acpiphp: Slot [22] registered [ 0.472141] acpiphp: Slot [23] registered [ 0.475288] acpiphp: Slot [24] registered [ 0.477255] acpiphp: Slot [25] registered [ 0.480163] acpiphp: Slot [26] registered [ 0.482158] acpiphp: Slot [27] registered [ 0.485159] acpiphp: Slot [28] registered [ 0.487497] acpiphp: Slot [29] registered [ 0.490362] acpiphp: Slot [30] registered [ 0.493678] acpiphp: Slot [31] registered [ 0.495214] PCI host bridge to bus 0000:00 [ 0.498039] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.503041] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.508302] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.513115] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.517059] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.521048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.524193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.528868] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.535086] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.546030] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.551069] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.555103] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.560029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.565028] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.567361] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.573171] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.576057] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.582152] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.588016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.600124] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.609024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.615478] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.638025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.652126] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.680028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.700000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.840034] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.848022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 1.024000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 1.033577] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 1.036971] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 1.044506] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 1.052512] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 1.056244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 1.065147] iommu: Default domain type: Passthrough [ 1.068563] SCSI subsystem initialized [ 1.071589] ACPI: bus type USB registered [ 1.073149] usbcore: registered new interface driver usbfs [ 1.076330] usbcore: registered new interface driver hub [ 1.080492] usbcore: registered new device driver usb [ 1.082446] pps_core: LinuxPPS API ver. 1 registered [ 1.086020] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 1.092122] PTP clock support registered [ 1.094463] EDAC MC: Ver: 3.0.0 [ 1.100234] PCI: Using ACPI for IRQ routing [ 1.122382] NetLabel: Initializing [ 1.124014] NetLabel: domain hash size = 128 [ 1.126012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 1.128105] NetLabel: unlabeled traffic allowed by default [ 1.131184] vgaarb: loaded [ 1.134112] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 1.137027] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.144844] clocksource: Switched to clocksource kvm-clock [ 1.351084] VFS: Disk quotas dquot_6.6.0 [ 1.354449] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.358897] *** VALIDATE ramfs *** [ 1.361134] *** VALIDATE hugetlbfs *** [ 1.362619] pnp: PnP ACPI init [ 1.368883] pnp: PnP ACPI: found 6 devices [ 1.402761] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.407506] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.410324] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.413733] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.416727] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.482787] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.484531] NET: Registered protocol family 2 [ 1.486757] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.499610] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.505766] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.515543] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.522651] TCP: Hash tables configured (established 65536 bind 65536) [ 1.526713] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.532816] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.543470] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.556462] NET: Registered protocol family 1 [ 1.562138] RPC: Registered named UNIX socket transport module. [ 1.566155] RPC: Registered udp transport module. [ 1.569450] RPC: Registered tcp transport module. [ 1.572304] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.577275] NET: Registered protocol family 44 [ 1.580659] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.588689] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.592714] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.597367] PCI: CLS 0 bytes, default 64 [ 1.601324] Unpacking initramfs... [ 4.053288] debug: unmapping init [mem 0xffff96823cc64000-0xffff96823ffcffff] [ 4.061771] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.067483] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.072876] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.192178] Initialise system trusted keyrings [ 5.194894] Key type blacklist registered [ 5.198140] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.214208] zbud: loaded [ 5.221096] *** VALIDATE nfs *** [ 5.222592] *** VALIDATE nfs4 *** [ 5.225628] pstore: using deflate compression [ 5.232489] Platform Keyring initialized [ 5.677763] NET: Registered protocol family 38 [ 5.680618] Key type asymmetric registered [ 5.709756] Asymmetric key parser 'x509' registered [ 5.712333] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.718754] io scheduler mq-deadline registered [ 5.720771] io scheduler kyber registered [ 5.723587] io scheduler bfq registered [ 5.726323] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.731224] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.735184] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.739024] ACPI: Power Button [PWRF] [ 5.908843] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.986385] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.005349] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.037878] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.088205] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.094625] Non-volatile memory driver v1.3 [ 6.096596] Linux agpgart interface v0.103 [ 6.157155] virtio_blk virtio1: [vda] 135128 512-byte logical blocks (69.2 MB/66.0 MiB) [ 6.160844] vda: detected capacity change from 0 to 69185536 [ 6.222623] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.227338] vdb: detected capacity change from 0 to 1073741824 [ 6.253026] libphy: Fixed MDIO Bus: probed [ 6.301357] usbcore: registered new interface driver usbserial_generic [ 6.305683] usbserial: USB Serial support registered for generic [ 6.309641] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.319171] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.322400] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.327575] mousedev: PS/2 mouse device common for all mice [ 6.335778] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.336711] rtc_cmos 00:05: RTC can wake from S4 [ 6.363254] rtc_cmos 00:05: registered as rtc0 [ 6.367958] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.369810] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.373543] intel_pstate: CPU model not supported [ 6.379670] hid: raw HID events driver (C) Jiri Kosina [ 6.389506] usbcore: registered new interface driver usbhid [ 6.391143] usbhid: USB HID core driver [ 6.392767] drop_monitor: Initializing network drop monitor service [ 6.395940] Initializing XFRM netlink socket [ 6.402091] NET: Registered protocol family 10 [ 6.413877] Segment Routing with IPv6 [ 6.415618] NET: Registered protocol family 17 [ 6.417126] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.420931] mpls_gso: MPLS GSO support [ 6.429222] RAS: Correctable Errors collector initialized. [ 6.430707] AVX version of gcm_enc/dec engaged. [ 6.432016] AES CTR mode by8 optimization enabled [ 6.833100] sched_clock: Marking stable (6833072632, 0)->(9180759206, -2347686574) [ 6.836451] registered taskstats version 1 [ 6.837807] Loading compiled-in X.509 certificates [ 6.839282] zswap: loaded using pool lzo/zbud [ 6.893070] Key type big_key registered [ 6.942966] Key type encrypted registered [ 6.946590] ima: No TPM chip found, activating TPM-bypass! [ 6.951604] ima: Allocated hash algorithm: sha1 [ 6.953964] ima: No architecture policies found [ 6.957747] evm: Initialising EVM extended attributes: [ 6.961607] evm: security.selinux [ 6.964571] evm: security.ima [ 6.966344] evm: security.capability [ 6.968543] evm: HMAC attrs: 0x1 [ 6.974343] rtc_cmos 00:05: setting system clock to 2026-03-10 00:17:20 UTC (1773101840) [ 6.991765] debug: unmapping init [mem 0xffffffffbc003000-0xffffffffbc1fffff] [ 6.999618] debug: unmapping init [mem 0xffffffffbad82000-0xffffffffbb058fff] [ 7.008888] Write protecting the kernel read-only data: 28672k [ 7.018978] debug: unmapping init [mem 0xffffffffb9403000-0xffffffffb95fffff] [ 7.026267] debug: unmapping init [mem 0xffffffffb9d14000-0xffffffffb9dfffff] [ 7.135187] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.257618] systemd[1]: Detected virtualization kvm. [ 7.265571] systemd[1]: Detected architecture x86-64. [ 7.270457] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.464561] systemd[1]: No hostname configured. [ 7.467296] systemd[1]: Set hostname to . [ 7.471424] random: systemd: uninitialized urandom read (16 bytes read) [ 7.475332] systemd[1]: Initializing machine ID from random generator. [ 7.731287] random: ln: uninitialized urandom read (6 bytes read) [ 8.173763] random: systemd: uninitialized urandom read (16 bytes read) [ 8.185427] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 8.204200] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 8.212414] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ 10.475732] hrtimer: interrupt took 2531587 ns [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.021747] device-mapper: uevent: version 1.0.3 [ 12.078885] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 14.629031] virtio_net virtio0 ens2: renamed from eth0 [ 14.891596] scsi host0: ata_piix [ 14.929670] scsi host1: ata_piix [ 14.988554] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 15.005230] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 17.590515] random: fast init done [ 20.480812] random: crng init done [ 20.490515] random: 7 urandom warning(s) missed due to ratelimiting [ 24.000813] dracut-initqueue[598]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 26.515303] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 30.106420] printk: systemd: 25 output lines suppressed due to ratelimiting [ 31.268362] SELinux: Disabled at runtime. [ 31.395538] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 31.412078] systemd[1]: Detected virtualization kvm. [ 31.415760] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 33.517707] systemd[1]: initrd-switch-root.service: Succeeded. [ 33.529326] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 33.542517] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 33.557857] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 33.571571] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 33.601678] systemd[1]: Starting Journal Service... Starting Journal Service... [ 33.626853] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ 33.765403] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Flush Journal to Persistent Storage... Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 35.872705] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 37.427212] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 37.727769] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 38.725271] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 38.820878] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit)[ 44.687616] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 45.750822] NFS: Registering the id_resolver key type [ 45.758991] Key type id_resolver registered [ 45.766884] Key type id_legacy registered [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg145-client login: [ 108.591104] libcfs: loading out-of-tree module taints kernel. [ 108.641670] Key type ._llcrypt registered [ 108.645791] Key type .llcrypt registered [ 109.255218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 109.271454] alg: No test for adler32 (adler32-zlib) [ 110.710946] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 111.469564] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 113.185285] Key type lgssc registered [ 114.797545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 219.226522] Lustre: Mounted lustre-client [ 223.217253] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 240.764792] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing check_logdir /tmp/testlogs/ [ 244.592465] Lustre: DEBUG MARKER: oleg145-client.virtnet: executing yml_node [ 244.704156] Lustre: lustre-OST0000-osc-ffff968298a7a800: disconnect after 24s idle [ 248.551466] Lustre: DEBUG MARKER: Client: 2.17.50.236 [ 250.481565] Lustre: DEBUG MARKER: MDS: 2.17.50.236 [ 252.300404] Lustre: DEBUG MARKER: OSS: 2.17.50.236 [ 253.534882] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Mar 9 20:21:25 EDT 2026 [ 265.518590] LustreError: 7916:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff968298a7a800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 265.543838] Lustre: Unmounted lustre-client [ 284.782472] Key type lgssc unregistered [ 285.029250] LNet: 8364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.034831] LNetError: 8364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 285.049810] LNet: Removed LNI 192.168.201.45@tcp [ 285.562167] Key type .llcrypt unregistered [ 285.564848] Key type ._llcrypt unregistered [ 306.516262] Key type ._llcrypt registered [ 306.519293] Key type .llcrypt registered [ 306.680693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 306.701884] alg: No test for adler32 (adler32-zlib) [ 307.576055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 307.641315] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 307.659409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 310.840321] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 310.912219] LNet: 9235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.915524] LNetError: 9235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 310.926760] LNet: Removed LNI 192.168.201.45@tcp [ 311.410840] Key type .llcrypt unregistered [ 311.413083] Key type ._llcrypt unregistered [ 315.332462] Key type ._llcrypt registered [ 315.334777] Key type .llcrypt registered [ 316.494830] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:22:28 (1773102148) [ 316.718365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 316.731672] alg: No test for adler32 (adler32-zlib) [ 317.627991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.703296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.726473] LNet: 9653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.750738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.786524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 317.800560] LNet: 9660:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 323.593787] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:22:35 (1773102155) [ 323.660684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.676474] LNet: 10230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.698306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.791700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 328.845142] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:22:41 (1773102161) [ 329.083453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.097428] LNet: 10822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.113124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.171873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 334.106186] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:22:46 (1773102166) [ 334.303765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.322113] LNet: 11406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.342125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.405017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 339.029353] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:22:51 (1773102171) [ 339.208859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 339.236245] LNet: 11991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 339.255768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.328441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 339.655370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 344.677728] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:22:57 (1773102177) [ 344.729078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.766520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.840422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 345.137689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 345.533664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 350.980823] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:23:03 (1773102183) [ 351.209114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.232033] LNet: 13179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.236824] LNet: 13179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 351.260355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.348412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 357.527592] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:23:09 (1773102189) [ 357.909243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.944712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.001246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 358.043201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 358.321944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 358.362448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 358.861745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 358.919800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 359.389450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 359.429607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 359.852896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 359.892760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.281183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 360.320162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 360.671493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 360.710880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.049330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 361.070583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 361.355338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 361.395158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 361.725472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 361.759902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 362.042177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 362.064989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 362.410695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 362.472396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 362.922598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 362.961346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 368.806471] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:23:21 (1773102201) [ 369.026521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.076429] LNet: 14446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.086089] LNet: 14446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 369.125798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.236958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 369.274816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 375.713935] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:23:28 (1773102208) [ 375.873959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.918098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.001203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 376.030564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 381.084071] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:23:33 (1773102213) [ 381.338893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.374900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.434109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 381.469301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 386.797591] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:23:39 (1773102219) [ 386.914722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.960445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.082212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 387.126674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 392.385073] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:23:44 (1773102224) [ 392.467552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.515130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.599682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 392.641095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 397.840046] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:23:50 (1773102230) [ 398.073528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.109307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.180372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 398.223515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 403.752846] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:23:56 (1773102236) [ 404.105921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.127083] LNet: 17964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.131892] LNet: 17964:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 404.163320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.226573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 404.592534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 409.976783] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:24:02 (1773102242) [ 410.187649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.217203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.400146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 410.606430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 416.140652] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:24:08 (1773102248) [ 416.234779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.281041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.392777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 416.578131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 421.825382] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:24:14 (1773102254) [ 422.027916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.058421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.437581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 422.684536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.876817] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:24:20 (1773102260) [ 428.074154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.114384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.193264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.257604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 433.428935] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:24:25 (1773102265) [ 433.523338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.564730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.626748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 433.669730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 438.608656] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:24:31 (1773102271) [ 438.697278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.735786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.821785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 443.363916] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:24:35 (1773102275) [ 443.540634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.579202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.663251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 448.714960] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:24:41 (1773102281) [ 448.885673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.915764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.995704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 449.286477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 455.199708] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:24:46 (1773102286) [ 455.314539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.362729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.436306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 455.493503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 460.761806] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:24:53 (1773102293) [ 461.064417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.106813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.174343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 461.205662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 466.161220] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:24:58 (1773102298) [ 466.328398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.388973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.599822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 466.638781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 471.428857] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:25:04 (1773102304) [ 471.546850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.569281] LNet: 25157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.573316] LNet: 25157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 471.609109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.671108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 471.718841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 476.751468] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:25:09 (1773102309) [ 476.849892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.891614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.940337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 476.994280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 477.010549] LNetError: 25758:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 477.047667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 477.084492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 481.817103] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:25:14 (1773102314) [ 481.932400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.977101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.040558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BX619wyYgw [ 482.104674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BX619wyYgw [ 482.144348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.BX619wyYgw [ 486.762955] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:25:19 (1773102319) [ 486.871408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.901728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.944161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 486.980537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 487.012107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 491.416305] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:25:24 (1773102324) [ 491.563383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.606679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.655553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 493.380298] LNet: Message forwarding will be enabled in 2s [ 496.481548] LNet: Message forwarding enabled [ 496.690148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 496.703564] LNet: Message forwarding will stop in 2s to 6s [ 501.083598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 501.537051] LNet: Message forwarding disabled [ 502.625132] LNet: Message forwarding will be enabled in 2s [ 506.656523] LNet: Message forwarding enabled [ 508.061594] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:25:40 (1773102340) [ 508.263651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.152302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.219291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 514.141608] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:25:46 (1773102346) [ 514.269957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.316838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.374737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 514.428328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 514.474429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 514.503605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 514.539339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 514.570417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 514.609061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 514.640617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 514.674398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 514.702372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 514.753837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 514.786986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 514.827216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 514.869342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 514.901295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 514.934754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 514.961990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 514.999630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 515.034446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 515.060453] LNetError: 28828:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 519.717846] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:25:52 (1773102352) [ 519.803690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.846840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.936916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 524.796549] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:25:57 (1773102357) [ 526.224643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg145-server [ 527.614235] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:26:00 (1773102360) [ 527.710240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.763413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.825271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 527.849566] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 529.253157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[46-50]@tcp [ 529.276995] LNet: 30543:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 529.306946] LNetError: 30544:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.49@tcp has gone from up to down [ 529.342646] LNetError: 30333:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.46@tcp is being used as a gateway but routing feature is not turned on [ 530.353935] LNetError: 30333:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.47@tcp is being used as a gateway but routing feature is not turned on [ 530.361574] LNetError: 30333:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 531.372086] LNetError: 30333:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.50@tcp is being used as a gateway but routing feature is not turned on [ 531.396543] LNetError: 30333:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 532.385064] LNetError: 30337:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.48@tcp has gone from up to down [ 532.397297] LNetError: 30337:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 533.417720] LNetError: 30333:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.46@tcp is being used as a gateway but routing feature is not turned on [ 533.428565] LNetError: 30333:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 534.703192] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:26:07 (1773102367) [ 536.334376] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg145-server [ 537.514794] LNetError: 30333:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.50@tcp is being used as a gateway but routing feature is not turned on [ 537.528602] LNetError: 30333:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 11 previous similar messages [ 537.719467] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:26:10 (1773102370) [ 537.870589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.528139] LNetError: 30332:0:(router.c:1262:lnet_check_routers()) Failed to discover router 192.168.201.46@tcp [ 538.914820] LNetError: 31477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 539.951039] LNet: Removed LNI 192.168.201.45@tcp [ 539.989808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.047229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 540.067364] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 540.622523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[46-51/2]@tcp [ 540.641404] LNet: 31595:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 540.651124] LNet: 31595:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 540.666941] LNetError: 31482:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.50@tcp has gone from up to down [ 540.668643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[46-51/2]@tcp [ 545.960380] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:26:18 (1773102378) [ 546.092482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.126777] LNetError: 32185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.141558] LNet: Removed LNI 192.168.201.45@tcp [ 546.159760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.216534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 546.262472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.293225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.348583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 546.408113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.441829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.475913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 546.538641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.571541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.616504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 546.681352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.716330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.764512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 546.831256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.865846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.952676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 547.043847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.091776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.138760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 547.209959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.239071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.283535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 547.370711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.401817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.466892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 547.535515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.575440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.659699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 547.734671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.782045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.828434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 547.905958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.942515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.997341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 552.966301] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:26:25 (1773102385) [ 554.393151] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg145-server [ 555.754130] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:26:28 (1773102388) [ 557.159785] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg145-server [ 558.876941] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:26:31 (1773102391) [ 559.560899] Key type .llcrypt unregistered [ 559.564302] Key type ._llcrypt unregistered [ 564.594215] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 564.790414] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 564.796896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 564.805468] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 567.353569] Key type ._llcrypt registered [ 567.358511] Key type .llcrypt registered [ 567.527834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 567.545100] alg: No test for adler32 (adler32-zlib) [ 568.426438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.468529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 568.493345] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 569.091876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 569.117667] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 570.697738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 570.727579] LNet: Removed LNI 10.1.2.3@tcp [ 570.847058] LNet: 34307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.858321] LNetError: 34307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 570.870952] LNet: Removed LNI 192.168.201.45@tcp [ 571.432609] Key type .llcrypt unregistered [ 571.436501] Key type ._llcrypt unregistered [ 580.617206] Key type ._llcrypt registered [ 580.627818] Key type .llcrypt registered [ 581.660947] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:26:54 (1773102414) [ 582.161317] Key type .llcrypt unregistered [ 582.163353] Key type ._llcrypt unregistered [ 587.176848] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 587.259127] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 587.262339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 587.266180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 589.755170] Key type ._llcrypt registered [ 589.758512] Key type .llcrypt registered [ 589.949829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 589.977679] alg: No test for adler32 (adler32-zlib) [ 590.846461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.895866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 590.928614] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 591.547876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 591.562297] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 593.589474] LNet: 35798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.594588] LNetError: 35798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.599757] LNet: Removed LNI 192.168.201.45@tcp [ 593.607906] LNet: Removed LNI 10.1.2.3@tcp [ 594.141959] Key type .llcrypt unregistered [ 594.143888] Key type ._llcrypt unregistered [ 603.059959] Key type ._llcrypt registered [ 603.065992] Key type .llcrypt registered [ 604.246573] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:27:16 (1773102436) [ 604.630961] Key type .llcrypt unregistered [ 604.633345] Key type ._llcrypt unregistered [ 609.098046] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 609.230647] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 609.235141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 609.242172] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 611.876875] Key type ._llcrypt registered [ 611.879284] Key type .llcrypt registered [ 612.070654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 612.091911] alg: No test for adler32 (adler32-zlib) [ 613.039293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.067911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 613.095149] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 614.658435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 614.673794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 616.572089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 616.595924] LNet: Removed LNI 10.1.2.3@tcp [ 616.611333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 616.661606] LNet: Removed LNI 10.1.2.31@tcp [ 616.679829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 616.697209] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 617.265646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 617.283705] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 618.119579] LNet: 37716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.126438] LNetError: 37716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 618.133159] LNet: Removed LNI 10.1.2.3@tcp [ 618.142381] LNet: Removed LNI 10.1.2.31@tcp [ 618.721792] Key type .llcrypt unregistered [ 618.725368] Key type ._llcrypt unregistered [ 627.646985] Key type ._llcrypt registered [ 627.648837] Key type .llcrypt registered [ 628.650849] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:27:41 (1773102461) [ 628.950942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 628.972298] alg: No test for adler32 (adler32-zlib) [ 629.868077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.927765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 629.954971] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 630.819098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.840407] LNet: 38668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.846907] LNetError: 38668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.868503] LNet: Removed LNI 192.168.201.45@tcp [ 630.913951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.982910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.45@tcp [ 631.029389] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 631.873591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.45@tcp [ 631.905703] LNet: Removed LNI 192.168.201.45@tcp [ 631.980953] LNet: 38804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.828619] Key type .llcrypt unregistered [ 632.833229] Key type ._llcrypt unregistered [ 643.720241] Key type ._llcrypt registered [ 643.721956] Key type .llcrypt registered [ 644.856243] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:27:57 (1773102477) [ 646.512198] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg145-server [ 647.867669] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:28:00 (1773102480) [ 648.356641] Key type .llcrypt unregistered [ 648.358905] Key type ._llcrypt unregistered [ 653.094544] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 653.188838] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 653.195655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 653.202243] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 655.885433] Key type ._llcrypt registered [ 655.894180] Key type .llcrypt registered [ 656.039690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 656.049538] alg: No test for adler32 (adler32-zlib) [ 656.929520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.986950] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 657.649595] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 659.520208] LNet: 40639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.525438] LNetError: 40639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 659.530738] LNet: Removed LNI 192.168.201.45@tcp1 [ 659.539962] LNet: Removed LNI 10.1.2.3@tcp2 [ 660.088547] Key type .llcrypt unregistered [ 660.090895] Key type ._llcrypt unregistered [ 669.113535] Key type ._llcrypt registered [ 669.115558] Key type .llcrypt registered [ 670.121469] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:28:22 (1773102502) [ 670.470562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 670.489316] alg: No test for adler32 (adler32-zlib) [ 671.349195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.387283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.403163] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 672.020883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.037459] LNet: 41609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.044252] LNetError: 41609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.056433] LNet: Removed LNI 192.168.201.45@tcp [ 672.080265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.124959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 672.147205] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 676.752518] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:28:29 (1773102509) [ 676.871021] LNet: 42201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.877524] LNetError: 42201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.893546] LNet: Removed LNI 192.168.201.45@tcp [ 677.276530] Key type .llcrypt unregistered [ 677.282114] Key type ._llcrypt unregistered [ 681.016881] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 681.182763] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 681.191992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 681.200262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 683.660390] Key type ._llcrypt registered [ 683.665222] Key type .llcrypt registered [ 683.841384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 683.851254] alg: No test for adler32 (adler32-zlib) [ 684.736300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.782785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 684.801750] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 685.453759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 685.469877] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 687.128349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.145655] LNet: 42941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.150874] LNetError: 42941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.154355] LNet: Removed LNI 192.168.201.45@tcp [ 687.170082] LNet: Removed LNI 10.1.2.3@tcp [ 687.187927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.249117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 687.278889] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 687.285204] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 687.681930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.700423] LNet: 42968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.706936] LNetError: 42968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.710714] LNetError: 42968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 687.714691] LNet: Removed LNI 192.168.201.45@tcp [ 687.726236] LNet: Removed LNI 10.1.2.3@tcp [ 687.746985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.805799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 687.828303] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 687.834264] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 688.116119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 688.140776] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 688.918601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.931866] LNet: 43131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.937199] LNetError: 43131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.941867] LNetError: 43131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 688.946575] LNet: Removed LNI 192.168.201.45@tcp [ 688.949418] LNet: Removed LNI 10.1.2.3@tcp [ 688.962772] LNet: Removed LNI 192.168.201.45@tcp2 [ 688.985453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.036212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 689.086795] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 689.093647] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 689.099603] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 694.689221] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:28:47 (1773102527) [ 694.833869] LNet: 43727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.840420] LNetError: 43727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.844086] LNetError: 43727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 694.848641] LNet: Removed LNI 192.168.201.45@tcp [ 694.851588] LNet: Removed LNI 10.1.2.3@tcp [ 694.863150] LNet: Removed LNI 192.168.201.45@tcp2 [ 695.487786] Key type .llcrypt unregistered [ 695.489171] Key type ._llcrypt unregistered [ 699.809970] Key type ._llcrypt registered [ 699.813487] Key type .llcrypt registered [ 699.966321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 699.980837] alg: No test for adler32 (adler32-zlib) [ 700.847719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.882075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.905700] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 701.505254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.520307] LNet: 44122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.525190] LNetError: 44122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.537931] LNet: Removed LNI 192.168.201.45@tcp [ 701.548857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.585961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 701.604275] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 701.901971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.928830] LNet: Removed LNI 192.168.201.45@tcp [ 701.942585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.987557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 702.011944] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 702.301819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.316974] LNet: 44182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.321162] LNet: 44182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 702.326644] LNetError: 44182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.333973] LNetError: 44182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 702.347452] LNet: Removed LNI 192.168.201.45@tcp [ 702.364679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.416402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 702.435377] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 702.703036] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 702.819744] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 702.824462] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 702.827429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 705.018515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.044337] LNet: 44261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.051194] LNetError: 44261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.063722] LNet: Removed LNI 192.168.201.45@tcp [ 705.077603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.129704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 705.150982] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 709.735988] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:29:02 (1773102542) [ 709.803418] LNet: 44850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.808966] LNetError: 44850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 709.825128] LNet: Removed LNI 192.168.201.45@tcp [ 710.362988] Key type .llcrypt unregistered [ 710.366207] Key type ._llcrypt unregistered [ 714.565670] Key type ._llcrypt registered [ 714.568691] Key type .llcrypt registered [ 714.699852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.715460] alg: No test for adler32 (adler32-zlib) [ 715.557543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.600950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 715.625102] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 716.195751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.209530] LNet: 45255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.213820] LNetError: 45255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.229404] LNet: Removed LNI 192.168.201.45@tcp [ 716.245874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.290557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 716.312266] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 720.362355] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:29:13 (1773102553) [ 720.448533] LNet: 45844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.453645] LNetError: 45844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.462897] LNet: Removed LNI 192.168.201.45@tcp [ 720.894245] Key type .llcrypt unregistered [ 720.899719] Key type ._llcrypt unregistered [ 724.274316] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.365321] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.370988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 724.378745] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.771705] Key type ._llcrypt registered [ 726.773470] Key type .llcrypt registered [ 726.891135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.901071] alg: No test for adler32 (adler32-zlib) [ 727.734543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.765374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.786571] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 728.371149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.384794] LNet: 46298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.389961] LNetError: 46298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.405356] LNet: Removed LNI 192.168.201.45@tcp [ 728.423566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.482323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 728.511115] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 733.251973] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:29:25 (1773102565) [ 733.368879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 733.384249] LNet: 46896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.390352] LNetError: 46896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 733.409884] LNet: Removed LNI 192.168.201.45@tcp [ 733.434786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.514899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 737.503733] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:29:30 (1773102570) [ 737.618324] LNet: 47473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.177379] Key type .llcrypt unregistered [ 738.180963] Key type ._llcrypt unregistered [ 742.292494] Key type ._llcrypt registered [ 742.298224] Key type .llcrypt registered [ 742.449542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.464236] alg: No test for adler32 (adler32-zlib) [ 743.319678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.365145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 743.388694] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 743.993244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.017715] LNet: 47875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.024989] LNetError: 47875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.042520] LNet: Removed LNI 192.168.201.45@tcp [ 744.073384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.122969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 744.147691] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 748.669245] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:29:41 (1773102581) [ 748.841773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.857294] LNet: 48467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.863129] LNetError: 48467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.873050] LNet: Removed LNI 192.168.201.45@tcp [ 748.897422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.952971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 753.332517] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:29:45 (1773102585) [ 754.376624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 755.619760] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:29:48 (1773102588) [ 756.560951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 757.624947] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:29:50 (1773102590) [ 758.635994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 759.772683] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:29:52 (1773102592) [ 760.883243] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 762.030478] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:29:54 (1773102594) [ 762.975406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 764.215022] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:29:56 (1773102596) [ 764.318183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.339304] LNet: 50517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.358876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.396901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 764.413587] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 764.960256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.45 [ 764.978853] LNet: 50651:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 764.989869] LNet: Removed LNI 192.168.201.45@tcp [ 769.024117] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:30:01 (1773102601) [ 769.147099] LNet: 51218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.153583] LNetError: 51218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.172990] LNet: Removed LNI 192.168.201.45@tcp [ 769.657745] Key type .llcrypt unregistered [ 769.663528] Key type ._llcrypt unregistered [ 773.007585] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 773.203898] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 773.208631] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 773.211928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 775.718177] Key type ._llcrypt registered [ 775.723086] Key type .llcrypt registered [ 775.839271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 775.849272] alg: No test for adler32 (adler32-zlib) [ 776.971750] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 781.850619] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:30:14 (1773102614) [ 781.969781] LNet: 52122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.978704] LNetError: 52122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 781.992577] LNet: Removed LNI 10.1.2.3@tcp [ 782.459061] Key type .llcrypt unregistered [ 782.463098] Key type ._llcrypt unregistered [ 786.493727] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 786.658354] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 786.661421] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 786.665279] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 789.297462] Key type ._llcrypt registered [ 789.301040] Key type .llcrypt registered [ 789.424077] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 789.443080] alg: No test for adler32 (adler32-zlib) [ 790.322855] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 794.630562] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:30:27 (1773102627) [ 794.753403] LNet: 53023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.759582] LNetError: 53023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 794.772357] LNet: Removed LNI 10.1.2.3@tcp [ 795.192876] Key type .llcrypt unregistered [ 795.197534] Key type ._llcrypt unregistered [ 799.007205] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 799.173300] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 799.176498] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 799.179603] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 801.653520] Key type ._llcrypt registered [ 801.655387] Key type .llcrypt registered [ 801.769070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 801.787433] alg: No test for adler32 (adler32-zlib) [ 802.706303] LNet: 53324:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 802.715032] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 807.081881] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:30:39 (1773102639) [ 807.136967] LNet: 53907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.141553] LNetError: 53907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 807.152714] LNet: Removed LNI 10.1.2.3@tcp [ 807.614519] Key type .llcrypt unregistered [ 807.620325] Key type ._llcrypt unregistered [ 811.633172] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 811.844069] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.849148] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.852138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 814.342271] Key type ._llcrypt registered [ 814.343518] Key type .llcrypt registered [ 814.464689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.475327] alg: No test for adler32 (adler32-zlib) [ 815.364633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.432493] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 822.109620] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:30:54 (1773102654) [ 822.451667] LNet: 55145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.457226] LNetError: 55145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 822.474232] LNet: Removed LNI 10.1.2.3@tcp [ 823.091111] Key type .llcrypt unregistered [ 823.095802] Key type ._llcrypt unregistered [ 827.078300] Key type ._llcrypt registered [ 827.081155] Key type .llcrypt registered [ 827.239201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.258612] alg: No test for adler32 (adler32-zlib) [ 828.175761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 828.207812] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 833.551233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 834.131864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_interrupt -r 1 [ 834.168739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_interrupt -r 1 [ 834.205745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 834.226936] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 834.400181] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 834.406627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 834.436421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 834.468914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 834.493204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 834.523879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 834.563984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 834.602983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 834.639112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 834.653348] LNet: 55581:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 834.679254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 834.718151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 834.754793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 834.792702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 834.824035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 834.861173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 834.896903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 835.339276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_dropped -r 1 [ 835.382588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_dropped -r 1 [ 835.438322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 835.456418] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 835.628430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 835.663506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 835.709308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 835.729741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 835.751516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 835.773918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 835.803371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 835.847908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 835.875778] LNet: 55691:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 835.892790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 835.918837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 835.945298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 835.966155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 835.986648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 836.020849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 836.050107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 836.379854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_aborted -r 1 [ 836.412658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_aborted -r 1 [ 836.441797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 836.460771] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 836.608636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 836.632443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 836.657327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 836.681281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 836.702507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 836.722866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 836.739896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 836.765993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 836.787096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 836.815276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 836.844796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 836.866663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 836.889305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 836.913868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 836.938842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 837.259175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_no_route -r 1 [ 837.286385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_no_route -r 1 [ 837.314868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 837.459508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 837.481123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 837.505291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 837.525730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 837.547914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 837.569287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 837.600137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 837.624232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 837.637901] LNet: 55909:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 837.642873] LNet: 55909:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 837.657212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 837.682141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 837.707614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 837.729086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 837.766688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 837.790529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 837.812415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 838.158147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_timeout -r 1 [ 838.193193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_timeout -r 1 [ 838.215709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 838.393868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 838.418222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 838.467475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 838.502413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 838.548118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 838.573069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 838.596599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 838.638871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 838.662333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 838.689130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 838.717112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 838.746688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 838.775913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.802071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 838.827802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 839.158038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_error -r 1 [ 839.189286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_error -r 1 [ 839.225138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 839.241634] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 839.245220] LNet: Skipped 3 previous similar messages [ 839.409303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 839.435813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 839.473909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 839.500465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 839.528626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 839.547831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 839.579377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 839.621323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 839.644026] LNet: 56127:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 839.652061] LNet: 56127:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 839.678777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 839.704257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 839.737910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 839.762787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 839.790513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.815974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.838792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 840.027884] LNet: 56168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.032151] LNetError: 56168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.046697] LNet: Removed LNI 192.168.201.45@tcp [ 840.376033] Key type .llcrypt unregistered [ 840.378625] Key type ._llcrypt unregistered [ 850.929589] Key type ._llcrypt registered [ 850.931662] Key type .llcrypt registered [ 851.829153] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:31:24 (1773102684) [ 852.524569] Key type .llcrypt unregistered [ 852.526641] Key type ._llcrypt unregistered [ 855.886275] Key type ._llcrypt registered [ 855.887372] Key type .llcrypt registered [ 856.034382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.051914] alg: No test for adler32 (adler32-zlib) [ 856.899307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 856.924134] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 861.675567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 868.758640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.791104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 868.804304] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 869.906261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_interrupt -r 1 [ 869.955589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_interrupt -r 1 [ 869.994664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_interrupt -r 1 [ 870.030682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_interrupt -r 1 [ 870.067700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_interrupt -r 1 [ 870.099454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_interrupt -r 1 [ 870.133207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_interrupt -r 1 [ 870.181975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_interrupt -r 1 [ 870.217144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 870.234789] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 870.244843] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 870.497243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 870.526200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 870.552750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 870.577665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 870.602441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 870.635795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 870.660651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 870.690807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 870.711216] LNet: 57633:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 870.739591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 870.765525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 870.801207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 870.821563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 870.844742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 870.869533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 870.896137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 871.353545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_dropped -r 1 [ 871.384556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_dropped -r 1 [ 871.427857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_dropped -r 1 [ 871.466908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_dropped -r 1 [ 871.488644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_dropped -r 1 [ 871.522140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_dropped -r 1 [ 871.571159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_dropped -r 1 [ 871.611290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_dropped -r 1 [ 871.647840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 871.671694] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 871.677427] LNet: Skipped 2 previous similar messages [ 871.917214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 871.952548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 871.979548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 872.014815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 872.050404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 872.098789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 872.133839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 872.176429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 872.193552] LNet: 57754:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 872.210575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 872.235059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 872.270371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 872.316774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 872.363244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 872.418477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 872.453813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 872.933118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_aborted -r 1 [ 872.997307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_aborted -r 1 [ 873.036422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_aborted -r 1 [ 873.067407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_aborted -r 1 [ 873.123114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_aborted -r 1 [ 873.192110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_aborted -r 1 [ 873.234615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_aborted -r 1 [ 873.275882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_aborted -r 1 [ 873.316273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 873.337213] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 873.344576] LNet: Skipped 2 previous similar messages [ 873.559719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 873.591429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 873.621789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 873.646619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 873.679323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 873.716741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 873.761820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 873.794144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 873.810354] LNet: 57876:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 873.824426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 873.859113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 873.907684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 873.942277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 873.981200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 874.027539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 874.074963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 874.542327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_no_route -r 1 [ 874.595164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_no_route -r 1 [ 874.644655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_no_route -r 1 [ 874.695613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_no_route -r 1 [ 874.749441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_no_route -r 1 [ 874.785724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_no_route -r 1 [ 874.825437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_no_route -r 1 [ 874.883232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_no_route -r 1 [ 874.928089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 875.164877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 875.200333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 875.232239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 875.264893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 875.297383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 875.324370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 875.360836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 875.390583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 875.418478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 875.448530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 875.492874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 875.527859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 875.573295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 875.617737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 875.657551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 876.242675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_timeout -r 1 [ 876.276237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_timeout -r 1 [ 876.306499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_timeout -r 1 [ 876.337878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_timeout -r 1 [ 876.382633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_timeout -r 1 [ 876.428409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_timeout -r 1 [ 876.477266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_timeout -r 1 [ 876.514275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_timeout -r 1 [ 876.557162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 876.578814] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 876.582868] LNet: Skipped 5 previous similar messages [ 876.755626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 876.786539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 876.812604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 876.832401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 876.851619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 876.878425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 876.913490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 876.952762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 876.970476] LNet: 58118:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 876.975111] LNet: 58118:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 876.996264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 877.027339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 877.058411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 877.088427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 877.118576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 877.148209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 877.172332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 877.637263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_error -r 1 [ 877.675297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_error -r 1 [ 877.716655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_error -r 1 [ 877.761647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_error -r 1 [ 877.801653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_error -r 1 [ 877.843368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_error -r 1 [ 877.874846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_error -r 1 [ 877.904861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_error -r 1 [ 877.941773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 878.135268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 878.166710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 878.200660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 878.239407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 878.272712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 878.316763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 878.351246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 878.394781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 878.428549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 878.458215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 878.492099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 878.522774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 878.560241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 878.599322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 878.652198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 882.885266] LNet: 58315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.896177] LNetError: 58315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.905046] LNet: Removed LNI 192.168.201.45@tcp [ 883.948946] LNet: Removed LNI 192.168.201.45@tcp1 [ 884.333542] Key type .llcrypt unregistered [ 884.338188] Key type ._llcrypt unregistered [ 895.701295] Key type ._llcrypt registered [ 895.705153] Key type .llcrypt registered [ 896.663995] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:32:09 (1773102729) [ 897.175528] Key type .llcrypt unregistered [ 897.177450] Key type ._llcrypt unregistered [ 901.176120] Key type ._llcrypt registered [ 901.181766] Key type .llcrypt registered [ 901.368110] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.378235] alg: No test for adler32 (adler32-zlib) [ 902.236952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 902.257837] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 907.530607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 908.024210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_dropped -r 1 [ 908.049709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_dropped -r 1 [ 908.074737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 908.088954] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 908.264980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 908.295993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 908.321265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 908.349252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 908.379441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 908.410791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 908.437657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 908.465741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 908.482414] LNet: 59562:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 908.496390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 908.527464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 908.553239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 908.574227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 908.597074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 908.624827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 908.653198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 909.059025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_error -r 1 [ 909.092985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_error -r 1 [ 909.128100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 909.148763] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 909.301971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 909.324658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 909.353186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 909.376418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 909.395706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 909.423512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 909.443673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 909.466808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 909.485906] LNet: 59671:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 909.501592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 909.529898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 909.554835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 909.579835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 909.602426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 909.623139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 909.649232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 910.019013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_timeout -r 1 [ 910.050850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_timeout -r 1 [ 910.080570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 910.251631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 910.279833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 910.310289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 910.336513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 910.357783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 910.390132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 910.428390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 910.467804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 910.489271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 910.518502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 910.555827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 910.578411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 910.606590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 910.631169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 910.657045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 910.856122] LNet: 59821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.864090] LNetError: 59821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.877691] LNet: Removed LNI 192.168.201.45@tcp [ 911.198237] Key type .llcrypt unregistered [ 911.202707] Key type ._llcrypt unregistered [ 922.753705] Key type ._llcrypt registered [ 922.758368] Key type .llcrypt registered [ 923.773741] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:32:36 (1773102756) [ 924.342338] Key type .llcrypt unregistered [ 924.344726] Key type ._llcrypt unregistered [ 928.722500] Key type ._llcrypt registered [ 928.724164] Key type .llcrypt registered [ 928.848218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.862733] alg: No test for adler32 (adler32-zlib) [ 929.743869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 929.772608] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 935.925519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 942.266153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.309725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 942.332261] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 943.652277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_dropped -r 1 [ 943.673739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e remote_dropped -r 1 [ 943.695926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_dropped -r 1 [ 943.722633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e remote_dropped -r 1 [ 943.757622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e remote_dropped -r 1 [ 943.793363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e remote_dropped -r 1 [ 943.829979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e remote_dropped -r 1 [ 943.860486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e remote_dropped -r 1 [ 943.889908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 943.905584] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 943.910501] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.145@tcp [ 944.102035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 944.126137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 944.156737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 944.203257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 944.248141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 944.277136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 944.317413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 944.351880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 944.372593] LNet: 61288:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 944.396603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 944.428408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 944.460505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 944.502074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 944.531490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 944.576405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 944.612600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 944.649439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 945.033322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_error -r 1 [ 945.060938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e remote_error -r 1 [ 945.086556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_error -r 1 [ 945.115300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e remote_error -r 1 [ 945.143454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e remote_error -r 1 [ 945.170329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e remote_error -r 1 [ 945.199786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e remote_error -r 1 [ 945.229581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e remote_error -r 1 [ 945.265357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 945.288328] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 945.294155] LNet: Skipped 2 previous similar messages [ 945.507155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 945.537839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 945.566868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 945.598085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 945.628654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 945.663376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 945.697101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 945.729703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 945.754328] LNet: 61411:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 945.771509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 945.809983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 945.847968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 945.868531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 945.895244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 945.926869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 945.949897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 945.978629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 946.441769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_timeout -r 1 [ 946.473576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e remote_timeout -r 1 [ 946.515524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_timeout -r 1 [ 946.553482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e remote_timeout -r 1 [ 946.590721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e remote_timeout -r 1 [ 946.622600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e remote_timeout -r 1 [ 946.654537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e remote_timeout -r 1 [ 946.689888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e remote_timeout -r 1 [ 946.724216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.145@tcp [ 946.739541] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 946.904060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 946.935308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 946.978968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 947.006276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 947.030923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 947.053339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 947.081089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 947.113311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 947.125246] LNet: 61535:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 947.141534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 947.175727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 947.198709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 947.223051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 947.243218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 947.264530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 947.290243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 947.311058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 950.736876] LNet: 61612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.742136] LNetError: 61612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.749264] LNet: Removed LNI 192.168.201.45@tcp [ 950.758978] LNet: Removed LNI 192.168.201.45@tcp1 [ 951.084488] Key type .llcrypt unregistered [ 951.086901] Key type ._llcrypt unregistered [ 962.802585] Key type ._llcrypt registered [ 962.807156] Key type .llcrypt registered [ 963.920992] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:33:16 (1773102796) [ 964.278563] Key type .llcrypt unregistered [ 964.281869] Key type ._llcrypt unregistered [ 968.282588] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 968.384927] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 968.387483] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 968.390709] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 970.840963] Key type ._llcrypt registered [ 970.842893] Key type .llcrypt registered [ 970.950871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 970.969527] alg: No test for adler32 (adler32-zlib) [ 971.881378] LNet: 62688:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 971.889707] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 971.925565] LNet: 62702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.931761] LNetError: 62702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.953691] LNet: Removed LNI 192.168.201.45@tcp [ 972.433499] Key type .llcrypt unregistered [ 972.437174] Key type ._llcrypt unregistered [ 976.666906] Key type ._llcrypt registered [ 976.669262] Key type .llcrypt registered [ 976.789140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.803197] alg: No test for adler32 (adler32-zlib) [ 977.670831] LNet: 62960:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 977.680607] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 977.683148] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 977.710134] LNet: 62974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.715252] LNetError: 62974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 977.721306] LNet: Removed LNI 192.168.201.45@tcp [ 977.735829] LNet: Removed LNI 10.1.2.3@tcp1 [ 978.215810] Key type .llcrypt unregistered [ 978.218090] Key type ._llcrypt unregistered [ 982.093599] Key type ._llcrypt registered [ 982.095179] Key type .llcrypt registered [ 982.224618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 982.240395] alg: No test for adler32 (adler32-zlib) [ 983.113843] LNet: 63233:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 983.123081] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 983.126856] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 983.160107] LNet: 63247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.168197] LNetError: 63247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.176289] LNet: Removed LNI 192.168.201.45@tcp [ 983.187707] LNet: Removed LNI 10.1.2.3@tcp [ 983.692435] Key type .llcrypt unregistered [ 983.694225] Key type ._llcrypt unregistered [ 987.314047] Key type ._llcrypt registered [ 987.316477] Key type .llcrypt registered [ 987.442833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 987.458347] alg: No test for adler32 (adler32-zlib) [ 988.336254] LNet: 63507:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 988.346922] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 988.351874] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 988.386849] LNet: 63521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.391258] LNetError: 63521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.396611] LNet: Removed LNI 192.168.201.45@tcp [ 988.409615] LNet: Removed LNI 10.1.2.3@tcp [ 988.861675] Key type .llcrypt unregistered [ 988.863748] Key type ._llcrypt unregistered [ 992.308627] Key type ._llcrypt registered [ 992.311290] Key type .llcrypt registered [ 992.442593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 992.456070] alg: No test for adler32 (adler32-zlib) [ 993.316782] LNet: 63786:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 993.326771] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 993.334131] LNetError: 63786:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 993.338793] LNetError: Error -100 starting up LNI tcp [ 993.351150] LNet: Removed LNI 192.168.201.45@tcp [ 993.798366] Key type .llcrypt unregistered [ 993.800611] Key type ._llcrypt unregistered [ 1000.689895] Key type ._llcrypt registered [ 1000.692116] Key type .llcrypt registered [ 1001.494283] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:33:54 (1773102834) [ 1002.091771] Key type .llcrypt unregistered [ 1002.093268] Key type ._llcrypt unregistered [ 1006.134964] Key type ._llcrypt registered [ 1006.140749] Key type .llcrypt registered [ 1006.307901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1006.322592] alg: No test for adler32 (adler32-zlib) [ 1007.207360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1007.232597] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1012.483403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1012.932699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e network_timeout -r 1 [ 1012.973599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e network_timeout -r 1 [ 1013.015141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1013.029438] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1013.159243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1013.178940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1013.202615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1013.228799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1013.254511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1013.273208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1013.296176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1013.314764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1013.332848] LNet: 65039:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1013.350749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1013.377652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1013.408171] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1013.416870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1013.451868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1013.477807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1013.504827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1013.699454] LNet: 65078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.705564] LNetError: 65078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.717143] LNet: Removed LNI 192.168.201.45@tcp [ 1013.984735] Key type .llcrypt unregistered [ 1013.988962] Key type ._llcrypt unregistered [ 1026.979589] Key type ._llcrypt registered [ 1026.985355] Key type .llcrypt registered [ 1027.180351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1027.192183] alg: No test for adler32 (adler32-zlib) [ 1028.044813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1028.065214] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1034.396168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1041.122748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1041.157824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1041.181855] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1042.380786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e network_timeout -r 1 [ 1042.412594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e network_timeout -r 1 [ 1042.442545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e network_timeout -r 1 [ 1042.485227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e network_timeout -r 1 [ 1042.522276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e network_timeout -r 1 [ 1042.554223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e network_timeout -r 1 [ 1042.581354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e network_timeout -r 1 [ 1042.612244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e network_timeout -r 1 [ 1042.649549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1042.667132] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1042.872211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.904738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1042.937947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.967867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1042.996948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.030215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.054649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.082368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.105754] LNet: 65957:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1043.119520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.150745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.179829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.206676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.237099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.260489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1046.679317] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1046.913802] LNet: 66030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.917965] LNetError: 66030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.924032] LNet: Removed LNI 192.168.201.45@tcp [ 1046.935177] LNet: Removed LNI 192.168.201.45@tcp1 [ 1047.264717] Key type .llcrypt unregistered [ 1047.267837] Key type ._llcrypt unregistered [ 1057.766305] Key type ._llcrypt registered [ 1057.769104] Key type .llcrypt registered [ 1058.686510] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:34:51 (1773102891) [ 1058.872558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1058.884960] alg: No test for adler32 (adler32-zlib) [ 1059.729709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.770572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1059.785533] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1060.213615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1060.224763] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1060.776461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1060.819747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1060.850718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1060.875147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1060.904399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1060.929332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1060.956630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.45@tcp [ 1060.970541] LNetError: 67130:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.45@tcp to recovery [ 1061.795233] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1061.795289] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1063.840932] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1064.270205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1067.936395] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1068.706974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1068.729160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1068.768677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1068.782774] LNet: 67247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.791236] LNetError: 67247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.796330] LNet: Removed LNI 192.168.201.45@tcp [ 1068.806618] LNet: Removed LNI 192.168.201.45@tcp1 [ 1068.820768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.859214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1068.880612] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1069.327115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1069.338660] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1069.911186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1069.933887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1069.970138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1070.008192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1070.041466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1070.069615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1070.089805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1070.110703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.45@tcp [ 1070.129193] LNetError: 67520:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.45@tcp to recovery [ 1070.880470] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1073.397081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1077.024622] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1077.031699] LNet: Skipped 1 previous similar message [ 1082.044853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1082.066456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1082.092540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.425895] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:35:19 (1773102919) [ 1086.498990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1086.512159] LNet: 68243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.516232] LNetError: 68243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.521379] LNetError: 68243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1086.524921] LNet: Removed LNI 192.168.201.45@tcp [ 1086.534384] LNet: Removed LNI 192.168.201.45@tcp1 [ 1086.550835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.585847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1086.605521] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1087.049431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1087.062208] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1087.580384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1087.626500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1087.649762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1087.678703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1087.700235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1087.725266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1087.748236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.45@tcp --health 0 [ 1087.799507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1088.608315] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.45@tcp [ 1088.608358] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1088.618638] LNet: Skipped 1 previous similar message [ 1103.657841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1103.754525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1103.799212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1103.826969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.45@tcp --health 500 [ 1103.897617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1109.231634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1109.277799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1109.296573] LNet: 68737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.304698] LNetError: 68737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.311994] LNetError: 68737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1109.316150] LNet: Removed LNI 192.168.201.45@tcp [ 1109.323586] LNet: Removed LNI 192.168.201.45@tcp1 [ 1109.336177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1109.373202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1109.395979] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1109.863401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1109.874196] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1110.421240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1110.449766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.491980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1110.510272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1110.537265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1110.574399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1110.600862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1110.640463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.45@tcp --health 0 [ 1111.392633] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1111.402172] LNet: Skipped 3 previous similar messages [ 1111.844873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1122.715725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1122.746565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.768823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.101537] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:35:59 (1773102959) [ 1127.496090] LNet: 69794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.503543] LNetError: 69794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.509441] LNetError: 69794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1127.516625] LNet: Removed LNI 192.168.201.45@tcp [ 1127.529734] LNet: Removed LNI 192.168.201.45@tcp1 [ 1128.049136] Key type .llcrypt unregistered [ 1128.054350] Key type ._llcrypt unregistered [ 1132.020293] Key type ._llcrypt registered [ 1132.023496] Key type .llcrypt registered [ 1132.187244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1132.210711] alg: No test for adler32 (adler32-zlib) [ 1133.123340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1133.144384] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1139.268070] Lustre: DEBUG MARKER: Initial discovery [ 1139.336693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.145@tcp [ 1140.693628] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1140.841824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1140.868526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.145@tcp [ 1140.891716] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1140.906749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1140.936691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.145@tcp -r 1 -m GET -l 3 [ 1140.971298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.145@tcp -r 1 -m GET -e local_error [ 1140.999995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.145@tcp -r 1 -m PUT -l 6 [ 1141.345573] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1141.844953] Lustre: DEBUG MARKER: Force oleg145-client.virtnet to discover 192.168.201.145@tcp (in background) [ 1141.988446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.145@tcp [ 1142.994626] Lustre: DEBUG MARKER: Force oleg145-server to discover 192.168.201.45@tcp [ 1144.770703] Lustre: DEBUG MARKER: Wait for 70556 [ 1145.056470] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1146.026185] Lustre: DEBUG MARKER: Finished wait on 70556 [ 1146.064882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1146.087718] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1146.095076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1146.247139] LNet: 71004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1146.252234] LNetError: 71004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1146.267358] LNet: Removed LNI 192.168.201.45@tcp [ 1146.534954] Key type .llcrypt unregistered [ 1146.538272] Key type ._llcrypt unregistered [ 1156.590526] Key type ._llcrypt registered [ 1156.595437] Key type .llcrypt registered [ 1157.609271] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:36:29 (1773102989) [ 1157.965349] Key type .llcrypt unregistered [ 1157.967525] Key type ._llcrypt unregistered [ 1162.192981] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1162.326386] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1162.333605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1162.343437] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1164.703553] Key type ._llcrypt registered [ 1164.707344] Key type .llcrypt registered [ 1164.879350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1164.893176] alg: No test for adler32 (adler32-zlib) [ 1165.803373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.848327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1165.866052] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1166.475518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1166.496442] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1172.808295] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:36:45 (1773103005) [ 1172.898152] LNet: 73072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.904133] LNetError: 73072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.911936] LNet: Removed LNI 192.168.201.45@tcp [ 1172.922090] LNet: Removed LNI 10.1.2.3@tcp [ 1173.370832] Key type .llcrypt unregistered [ 1173.373227] Key type ._llcrypt unregistered [ 1177.025834] Key type ._llcrypt registered [ 1177.027243] Key type .llcrypt registered [ 1177.135804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1177.144562] alg: No test for adler32 (adler32-zlib) [ 1178.001578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.043485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1178.065548] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1178.541718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1178.553851] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1179.262805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.45@tcp [ 1180.320120] LNetError: Unexpected error -22 connecting to 192.168.201.45@tcp at host 192.168.201.45:988 [ 1183.381175] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:36:55 (1773103015) [ 1183.442411] LNet: 74198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.452849] LNetError: 74198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.455376] LNet: Removed LNI 192.168.201.45@tcp [ 1183.460854] LNet: Removed LNI 10.1.2.3@tcp [ 1183.937514] Key type .llcrypt unregistered [ 1183.939444] Key type ._llcrypt unregistered [ 1187.629765] Key type ._llcrypt registered [ 1187.631507] Key type .llcrypt registered [ 1187.775494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1187.801253] alg: No test for adler32 (adler32-zlib) [ 1188.646134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1188.694993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1188.713200] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1189.195351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1189.209505] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1189.790368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.45@tcp1 --nid 192.168.201.45@tcp2 [ 1192.685928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp1 192.168.201.45@tcp2 [ 1192.703020] LNetError: 75040:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.45@tcp1 to 192.168.201.45@tcp2 - Route table may be misconfigured [ 1192.719180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp2 192.168.201.45@tcp1 [ 1196.894936] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:37:09 (1773103029) [ 1197.028753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1197.045605] LNet: 75613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.050778] LNetError: 75613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.057093] LNet: Removed LNI 192.168.201.45@tcp1 [ 1198.118937] LNet: Removed LNI 192.168.201.45@tcp2 [ 1198.144632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1198.183624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1198.204661] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1198.690728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1198.703293] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1199.298121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1199.327145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.45@tcp [ 1199.349665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp1 192.168.201.45@tcp1 [ 1199.370531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.45@tcp -d 192.168.201.45@tcp -e network_timeout [ 1199.397161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -e network_timeout [ 1199.424451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -e network_timeout [ 1199.459801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -e network_timeout [ 1199.485617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1199.500844] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1199.515460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1199.578673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1199.645307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1200.225279] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1203.919587] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:37:16 (1773103036) [ 1203.995668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1204.011212] LNet: 76471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1204.016298] LNetError: 76471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.021586] LNetError: 76471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1204.027048] LNet: Removed LNI 192.168.201.45@tcp [ 1205.094594] LNet: Removed LNI 192.168.201.45@tcp1 [ 1205.105888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1205.156817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1205.309559] LNet: 76511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.553593] Key type .llcrypt unregistered [ 1205.556441] Key type ._llcrypt unregistered [ 1211.733224] Key type ._llcrypt registered [ 1211.734663] Key type .llcrypt registered [ 1212.472824] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:37:25 (1773103045) [ 1212.662382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1212.669895] alg: No test for adler32 (adler32-zlib) [ 1213.529991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1214.430894] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1215.377368] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:37:28 (1773103048) [ 1215.448894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1215.469122] LNet: 77607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.483828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1215.525172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1215.541242] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1215.964819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1215.979601] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1216.517755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1216.540215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp1 [ 1216.563878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp1 [ 1220.334894] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:37:33 (1773103053) [ 1221.449208] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg145-server [ 1222.606081] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:37:35 (1773103055) [ 1223.893936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg145-server [ 1225.177855] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:37:37 (1773103057) [ 1226.300146] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg145-server [ 1227.371435] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:37:40 (1773103060) [ 1228.599699] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg145-server [ 1229.649401] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:37:42 (1773103062) [ 1230.824033] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg145-server [ 1232.020748] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:37:44 (1773103064) [ 1233.088544] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg145-server [ 1234.134436] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:37:46 (1773103066) [ 1235.329662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg145-server [ 1236.456991] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:37:49 (1773103069) [ 1237.714272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg145-server [ 1238.939747] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:37:51 (1773103071) [ 1240.261716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg145-server [ 1241.365745] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:37:53 (1773103073) [ 1241.515257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1241.532661] LNet: 81481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.536791] LNetError: 81481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.540777] LNet: Removed LNI 192.168.201.45@tcp [ 1242.605165] LNet: Removed LNI 192.168.201.45@tcp1 [ 1242.626287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1242.687679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1242.712144] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1243.133373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1243.188533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1243.309874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1243.325663] LNet: 81629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.330893] LNetError: 81629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.334419] LNetError: 81629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1243.348529] LNet: Removed LNI 192.168.201.45@tcp [ 1243.365676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.405781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1243.426566] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1243.900518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1244.010464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1245.098360] LNet: Removed LNI 192.168.201.45@tcp [ 1245.137738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1245.182301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1245.202305] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1245.643345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1245.695840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1245.832338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1245.847889] LNet: 81921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1245.852954] LNet: 81921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1245.857403] LNetError: 81921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1245.863169] LNetError: 81921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1245.877529] LNet: Removed LNI 192.168.201.45@tcp [ 1245.891684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1245.930356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1245.949710] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1246.434949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1247.641872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1248.679152] LNet: Removed LNI 192.168.201.45@tcp [ 1248.695532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1248.734427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1248.756332] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1249.191823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1249.233332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1249.383240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1249.397432] LNetError: 82223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1249.400526] LNetError: 82223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1249.410544] LNet: Removed LNI 192.168.201.45@tcp [ 1249.424868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1249.463068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1249.489714] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1249.966682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1250.108989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1250.122335] LNet: 82368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.125572] LNet: 82368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1251.176067] LNet: Removed LNI 192.168.201.45@tcp [ 1251.196241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1251.247687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1251.271191] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1251.788940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1251.846325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1253.080768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1254.118517] LNet: Removed LNI 192.168.201.45@tcp [ 1254.133865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1254.181234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1254.211820] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1254.808846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1254.987525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1255.003192] LNetError: 82670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.011629] LNetError: 82670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1255.027422] LNet: Removed LNI 192.168.201.45@tcp [ 1255.053272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1255.097185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1255.117860] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1255.731880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1255.788741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1255.949828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1256.999202] LNet: Removed LNI 192.168.201.45@tcp [ 1257.033409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1257.096200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1257.114718] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1257.687211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1258.928434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1258.943059] LNet: 82971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.947204] LNet: 82971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1260.007492] LNet: Removed LNI 192.168.201.45@tcp [ 1260.027373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1260.071189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1260.097619] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1260.566511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1260.621098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1260.772749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1261.798466] LNet: Removed LNI 192.168.201.45@tcp [ 1261.811830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1261.856961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1261.878653] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1262.390426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1264.731778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1264.759247] LNetError: 83282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.766817] LNetError: 83282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1264.786760] LNet: Removed LNI 192.168.201.45@tcp [ 1264.803802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1264.861804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1264.882918] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1265.398779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1265.439173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1265.463562] LNet: Removed LNI 192.168.201.45@tcp [ 1265.475769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1265.512732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1265.534473] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1266.004250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1270.213796] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:38:22 (1773103102) [ 1270.362338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1270.379039] LNet: Removed LNI 192.168.201.45@tcp [ 1270.390789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1270.430155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1270.455087] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/180] [ 1271.162773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1271.182782] LNet: Removed LNI 192.168.201.45@tcp231 [ 1271.198330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1271.247367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1271.270228] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/180] [ 1271.278919] LNet: 84264:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1271.525670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1271.544357] LNet: Removed LNI 192.168.201.45@tcp231 [ 1271.554328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1271.576785] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/180] [ 1272.305382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1272.325099] LNet: Removed LNI 192.168.201.45@tcp231 [ 1272.336741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.385463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1272.406069] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/0] [ 1272.410342] LNet: 84416:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1272.625949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1272.648851] LNet: Removed LNI 192.168.201.45@tcp231 [ 1272.659297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1272.678585] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/0] [ 1273.293717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1273.313413] LNet: Removed LNI 192.168.201.45@tcp231 [ 1273.326985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1273.368045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1273.387237] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/60] [ 1273.611470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1273.633613] LNet: Removed LNI 192.168.201.45@tcp231 [ 1273.648246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1273.661833] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/60] [ 1274.291134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1274.317487] LNet: Removed LNI 192.168.201.45@tcp231 [ 1274.330268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1274.376776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1274.392196] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/180] [ 1274.395487] LNet: 84718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1274.400335] LNet: 84718:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1274.621506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1274.656329] LNet: Removed LNI 192.168.201.45@tcp231 [ 1274.669039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1274.688397] LNet: Added LNI 192.168.201.45@tcp231 [8/256/0/180] [ 1278.840373] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:38:31 (1773103111) [ 1278.924573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1278.944864] LNet: 85427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.949510] LNet: 85427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1278.961691] LNet: Removed LNI 192.168.201.45@tcp231 [ 1278.975089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1279.021704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1279.042486] LNet: Added LNI 192.168.201.45@tcp232 [8/256/0/180] [ 1279.462749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1279.487034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1279.509418] LNet: Removed LNI 192.168.201.45@tcp232 [ 1279.522431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1279.538537] LNet: Added LNI 192.168.201.45@tcp232 [8/256/0/180] [ 1279.975164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1280.004387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1280.036806] LNet: Removed LNI 192.168.201.45@tcp232 [ 1280.048766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1280.066350] LNet: Added LNI 192.168.201.45@tcp232 [8/256/0/180] [ 1280.512470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1280.539609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1280.565751] LNet: Removed LNI 192.168.201.45@tcp232 [ 1280.577832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1280.597786] LNet: Added LNI 192.168.201.45@tcp232 [8/256/0/180] [ 1281.049460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1281.077385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1281.095524] LNet: Removed LNI 192.168.201.45@tcp232 [ 1281.109845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1284.708261] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:38:37 (1773103117) [ 1285.353821] Key type .llcrypt unregistered [ 1285.356074] Key type ._llcrypt unregistered [ 1288.879378] Key type ._llcrypt registered [ 1288.881603] Key type .llcrypt registered [ 1289.000763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1289.014724] alg: No test for adler32 (adler32-zlib) [ 1289.863445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1289.881760] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1294.424865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1299.920471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1299.954902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1299.973591] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1300.950957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_interrupt -r 2 [ 1300.977845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_interrupt -r 2 [ 1301.013112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_interrupt -r 2 [ 1301.057723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_interrupt -r 2 [ 1301.092233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_interrupt -r 2 [ 1301.130994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_interrupt -r 2 [ 1301.160205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_interrupt -r 2 [ 1301.185063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_interrupt -r 2 [ 1301.211377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1301.227377] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1301.232648] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1301.378091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1301.396812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1301.421713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1301.446914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1301.471109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1301.496110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1301.526044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1301.559465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1301.580724] LNet: 87215:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1301.601195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1301.628387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1301.654627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1301.675585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1301.699466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1301.722286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.036466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_dropped -r 2 [ 1302.057698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_dropped -r 2 [ 1302.079170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_dropped -r 2 [ 1302.112974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_dropped -r 2 [ 1302.152200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_dropped -r 2 [ 1302.180397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_dropped -r 2 [ 1302.205822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_dropped -r 2 [ 1302.227106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_dropped -r 2 [ 1302.249296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1302.261854] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1302.407227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1302.430659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1302.452684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1302.470506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1302.492943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.517438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.540123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.565554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.582714] LNet: 87335:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.603185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.625317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.644915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.672283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1302.698079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.716248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1303.042402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_aborted -r 2 [ 1303.070284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_aborted -r 2 [ 1303.106728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_aborted -r 2 [ 1303.135765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_aborted -r 2 [ 1303.162515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_aborted -r 2 [ 1303.197396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_aborted -r 2 [ 1303.229864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_aborted -r 2 [ 1303.263252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_aborted -r 2 [ 1303.294441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1303.313989] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1303.459638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1303.483059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1303.513090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1303.539786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1303.563684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1303.585604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1303.608573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1303.630809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1303.642836] LNet: 87455:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1303.661383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1303.681628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1303.710146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1303.724968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1303.745731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1303.766748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1304.068536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_no_route -r 2 [ 1304.095445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_no_route -r 2 [ 1304.116646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_no_route -r 2 [ 1304.142585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_no_route -r 2 [ 1304.166833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_no_route -r 2 [ 1304.196975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_no_route -r 2 [ 1304.232754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_no_route -r 2 [ 1304.260738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_no_route -r 2 [ 1304.292923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1304.454665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.476511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1304.500664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.525259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.547251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.568325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.593800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.628933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.654978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.688357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.712435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.737434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.772981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.806803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.213191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e local_timeout -r 2 [ 1305.237832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e local_timeout -r 2 [ 1305.269705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e local_timeout -r 2 [ 1305.300536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e local_timeout -r 2 [ 1305.331000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e local_timeout -r 2 [ 1305.368275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e local_timeout -r 2 [ 1305.401181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e local_timeout -r 2 [ 1305.429706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e local_timeout -r 2 [ 1305.456477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1305.470399] LNet: There was an unexpected network error while writing to 192.168.201.145: rc = -22 [ 1305.474177] LNet: Skipped 2 previous similar messages [ 1305.639969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.660839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1305.689387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.714203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.745715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.769398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.801907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.830134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.844456] LNet: 87696:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.849424] LNet: 87696:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1305.862508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.889811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.918549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.953043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.975155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.001395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.363363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp -m GET -e remote_dropped -r 2 [ 1306.403963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.145@tcp1 -m GET -e remote_dropped -r 2 [ 1306.441090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp -m GET -e remote_dropped -r 2 [ 1306.480068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp -d 192.168.201.45@tcp1 -m GET -e remote_dropped -r 2 [ 1306.515325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp -m GET -e remote_dropped -r 2 [ 1306.543365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.145@tcp1 -m GET -e remote_dropped -r 2 [ 1306.573133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp -m GET -e remote_dropped -r 2 [ 1306.611761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.45@tcp1 -d 192.168.201.45@tcp1 -m GET -e remote_dropped -r 2 [ 1306.641869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.45@tcp 192.168.201.145@tcp [ 1306.661679] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.145@tcp [ 1306.851919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.872912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1306.896797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1306.927362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1306.964785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1306.997550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.027179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.056591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.079107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.098876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.128747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.158412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.195343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.224753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1310.209574] LNet: 87889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.214926] LNetError: 87889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.221464] LNet: Removed LNI 192.168.201.45@tcp [ 1310.233440] LNet: Removed LNI 192.168.201.45@tcp1 [ 1310.482680] Key type .llcrypt unregistered [ 1310.484179] Key type ._llcrypt unregistered [ 1319.450675] Key type ._llcrypt registered [ 1319.453547] Key type .llcrypt registered [ 1320.220973] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:39:12 (1773103152) [ 1320.427761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.439563] alg: No test for adler32 (adler32-zlib) [ 1321.292982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1321.326344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1321.343368] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1321.872179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1321.896016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.45@tcp [ 1321.909186] LNetError: 88867:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.45@tcp to recovery [ 1321.965755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.45@tcp [ 1322.027367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1326.061930] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:39:18 (1773103158) [ 1326.217379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1326.235155] LNet: 89456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.243144] LNetError: 89456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.262952] LNet: Removed LNI 192.168.201.45@tcp [ 1326.277746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1326.319854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1326.335968] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1326.778116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1326.789637] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1327.387772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.45@tcp [ 1327.402815] LNetError: 89722:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.45@tcp to recovery [ 1328.352365] LNet: 1 local NIs in recovery (showing 1): 192.168.201.45@tcp [ 1328.354286] LNet: There was an unexpected network error while writing to 192.168.201.45: rc = -22 [ 1328.557378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.45@tcp [ 1334.284657] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:39:26 (1773103166) [ 1335.353444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg145-server [ 1336.324783] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 20:39:29 (1773103169) [ 1337.476107] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg145-server [ 1338.480873] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 20:39:31 (1773103171) [ 1339.626257] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg145-server [ 1340.533849] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:39:33 (1773103173) [ 1340.622624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1340.638332] LNet: 91350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.652905] LNetError: 91350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.658658] LNet: Removed LNI 192.168.201.45@tcp [ 1340.667505] LNet: Removed LNI 192.168.201.45@tcp1 [ 1340.683483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.712841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1340.729780] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1341.216671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1341.251611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1341.282364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1341.340419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1341.373556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1341.404043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1345.114026] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:39:37 (1773103177) [ 1345.171199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1345.182561] LNet: 92070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.185425] LNetError: 92070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.188293] LNetError: 92070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1345.195557] LNet: Removed LNI 192.168.201.45@tcp [ 1345.208162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.247433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1345.263408] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1349.204269] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:39:41 (1773103181) [ 1349.275336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1349.292781] LNet: 92775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.302352] LNetError: 92775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.331294] LNet: Removed LNI 192.168.201.45@tcp [ 1349.344941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.377400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1349.398721] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1353.655220] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:39:46 (1773103186) [ 1354.604444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1355.595855] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:39:48 (1773103188) [ 1355.848820] LNet: 93812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.854636] LNetError: 93812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.863497] LNet: Removed LNI 192.168.201.45@tcp [ 1356.269541] Key type .llcrypt unregistered [ 1356.271401] Key type ._llcrypt unregistered [ 1359.557865] Key type ._llcrypt registered [ 1359.559849] Key type .llcrypt registered [ 1359.689772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1359.707710] alg: No test for adler32 (adler32-zlib) [ 1360.547356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1360.564419] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1364.920187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1367.187445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.145@tcp [ 1367.409878] LNet: 94228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.423808] LNetError: 94228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.436304] LNet: Removed LNI 192.168.201.45@tcp [ 1367.694478] Key type .llcrypt unregistered [ 1367.697985] Key type ._llcrypt unregistered [ 1373.383910] Key type ._llcrypt registered [ 1373.386018] Key type .llcrypt registered [ 1374.156306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1375.049816] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1375.933219] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:40:08 (1773103208) [ 1376.144319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1376.156231] alg: No test for adler32 (adler32-zlib) [ 1376.990968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1377.038347] LNet: 95315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.364937] Key type .llcrypt unregistered [ 1377.366341] Key type ._llcrypt unregistered [ 1380.216562] Key type ._llcrypt registered [ 1380.217593] Key type .llcrypt registered [ 1380.328906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1380.341670] alg: No test for adler32 (adler32-zlib) [ 1381.190506] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1381.194467] LNet: 95571:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1381.202348] LNet: 95571:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1381.232471] LNetError: 95583:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.47@tcp is being used as a gateway but routing feature is not turned on [ 1381.251749] LNetError: 95583:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.201.47@tcp has gone from up to down [ 1381.533649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1384.608131] LNetError: 95583:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.201.48@tcp has gone from up to down [ 1384.613423] LNetError: 95583:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 1384.618553] LNet: 95629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.624505] LNetError: 95629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.649554] LNet: Removed LNI 192.168.201.45@tcp [ 1384.663770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1384.700090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1384.713824] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1384.718556] LNet: 95636:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1384.723482] LNet: 95636:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1384.727980] LNetError: 95636:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.201.47@tcp is a gateway. Can not delete it [ 1384.732893] LNetError: 95636:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.201.47@tcp is a router. Can not be deleted [ 1384.746306] LNetError: 95633:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.49@tcp is being used as a gateway but routing feature is not turned on [ 1384.754954] LNetError: 95633:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 1384.761765] LNet: 95636:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1387.361311] LNetError: 95647:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.201.48@tcp has gone from up to down [ 1387.368497] LNetError: 95647:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 1388.275205] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:40:20 (1773103220) [ 1389.290387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg145-server [ 1390.233354] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:40:22 (1773103222) [ 1391.178575] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg145-server [ 1392.006715] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:40:24 (1773103224) [ 1392.879617] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1394.609972] Key type lgssc registered [ 1395.387348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1406.797095] Key type lgssc unregistered [ 1407.039504] LNet: 98055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.046972] LNetError: 98055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.075532] LNet: Removed LNI 192.168.201.45@tcp [ 1407.479369] Key type .llcrypt unregistered [ 1407.481632] Key type ._llcrypt unregistered [ 1415.155378] Key type ._llcrypt registered [ 1415.157267] Key type .llcrypt registered [ 1415.895288] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:40:48 (1773103248) [ 1416.093522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1416.102306] alg: No test for adler32 (adler32-zlib) [ 1416.947124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.993136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1417.026651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1417.054419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1417.082261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1421.337281] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:40:53 (1773103253) [ 1421.407231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1421.425220] LNet: 99452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.443168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.498956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1421.522751] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1422.114299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.127410] LNet: 99604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.132979] LNetError: 99604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.145811] LNet: Removed LNI 192.168.201.45@tcp [ 1422.165384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.202538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1422.219785] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1422.491373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.519557] LNet: Removed LNI 192.168.201.45@tcp [ 1422.537850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.592611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1422.620416] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1427.374898] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:40:59 (1773103259) [ 1427.432488] LNet: 100214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.436813] LNet: 100214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1427.442124] LNetError: 100214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.445856] LNetError: 100214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1427.455274] LNet: Removed LNI 192.168.201.45@tcp [ 1427.908859] Key type .llcrypt unregistered [ 1427.911578] Key type ._llcrypt unregistered [ 1431.355063] Key type ._llcrypt registered [ 1431.363267] Key type .llcrypt registered [ 1431.522991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1431.536344] alg: No test for adler32 (adler32-zlib) [ 1432.378350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.429908] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1432.434465] LNet: *** cfs_fail_loc=e003, val=100*** [ 1432.873346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1432.888591] Ping reply received with mapping: [ 1432.890723] found metadata reply NI[192.168.201.45@tcp] [ 1432.893013] 2d c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1432.894883] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1432.896773] 5a 5a 5a 5a 5a 5a 5a 5a [ 1432.970287] LNet: 100619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1432.975544] LNetError: 100619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1432.990659] LNet: Removed LNI 192.168.201.45@tcp [ 1433.376666] Key type .llcrypt unregistered [ 1433.377919] Key type ._llcrypt unregistered [ 1439.218244] Key type ._llcrypt registered [ 1439.220335] Key type .llcrypt registered [ 1439.934412] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:41:12 (1773103272) [ 1440.315520] Key type .llcrypt unregistered [ 1440.318837] Key type ._llcrypt unregistered [ 1443.550777] Key type ._llcrypt registered [ 1443.552856] Key type .llcrypt registered [ 1443.662701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1443.670897] alg: No test for adler32 (adler32-zlib) [ 1444.679449] LNetError: 101663:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1444.938835] Key type .llcrypt unregistered [ 1444.941558] Key type ._llcrypt unregistered [ 1447.784681] Key type ._llcrypt registered [ 1447.786853] Key type .llcrypt registered [ 1448.511297] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:41:21 (1773103281) [ 1448.803458] Key type .llcrypt unregistered [ 1448.805291] Key type ._llcrypt unregistered [ 1451.788775] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1451.794304] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1454.092617] Key type ._llcrypt registered [ 1454.094479] Key type .llcrypt registered [ 1454.209763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1454.219405] alg: No test for adler32 (adler32-zlib) [ 1455.080157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.129047] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1461.052057] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:41:33 (1773103293) [ 1461.102030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.118344] LNet: 103437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.123352] LNetError: 103437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1461.133870] LNet: Removed LNI 10.1.2.3@tcp [ 1461.148794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.598446] Key type .llcrypt unregistered [ 1461.600829] Key type ._llcrypt unregistered [ 1464.697545] Key type ._llcrypt registered [ 1464.701324] Key type .llcrypt registered [ 1464.811334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1464.820607] alg: No test for adler32 (adler32-zlib) [ 1465.701074] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1465.760473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1466.919689] LNet: 103741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.924983] LNetError: 103741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.937578] LNet: Removed LNI 192.168.201.45@tcp [ 1467.321514] Key type .llcrypt unregistered [ 1467.323508] Key type ._llcrypt unregistered [ 1473.140490] Key type ._llcrypt registered [ 1473.142856] Key type .llcrypt registered [ 1473.939848] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:41:46 (1773103306) [ 1474.325475] Key type .llcrypt unregistered [ 1474.327579] Key type ._llcrypt unregistered [ 1477.635445] Key type ._llcrypt registered [ 1477.637678] Key type .llcrypt registered [ 1477.763271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1477.772800] alg: No test for adler32 (adler32-zlib) [ 1482.311517] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:41:54 (1773103314) [ 1482.376477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.403209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.442789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1482.462463] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1482.987849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.008676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1483.031716] LNet: Removed LNI 192.168.201.45@tcp [ 1483.176071] LNet: 105483:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1483.406428] Key type .llcrypt unregistered [ 1483.408131] Key type ._llcrypt unregistered [ 1488.916203] Key type ._llcrypt registered [ 1488.917630] Key type .llcrypt registered [ 1489.658548] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:42:02 (1773103322) [ 1489.869166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1489.880608] alg: No test for adler32 (adler32-zlib) [ 1490.736607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.767546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1490.781814] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1491.283587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1491.318805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.45@tcp [ 1491.328263] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1491.331594] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1491.334532] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1491, 0, 0 [ 1491.337614] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1491.339766] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1491.343317] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1491.345591] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1491, 0, 0 [ 1491.349494] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1491.352034] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1491.354833] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1491.357469] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1491, 0, 1541 [ 1491.360459] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1491.362558] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1491.364798] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1491.367408] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1491, 0, 1541 [ 1491.371889] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1494.874404] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:42:07 (1773103327) [ 1495.167547] LNet: 107020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.174515] LNetError: 107020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.187154] LNet: Removed LNI 192.168.201.45@tcp [ 1495.527787] Key type .llcrypt unregistered [ 1495.530092] Key type ._llcrypt unregistered [ 1498.690093] Key type ._llcrypt registered [ 1498.691904] Key type .llcrypt registered [ 1498.827635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1498.834605] alg: No test for adler32 (adler32-zlib) [ 1499.672395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1499.687960] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1503.925913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1507.779769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.810368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1507.836586] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1508.412790] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1508.480916] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1508.483432] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1508.486160] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1510.650585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1510.663754] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1516.626212] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:42:29 (1773103349) [ 1516.668303] LNet: 108519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.672402] LNetError: 108519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.676944] LNet: Removed LNI 192.168.201.45@tcp [ 1517.728335] LNet: Removed LNI 192.168.201.45@tcp1 [ 1517.730857] LNetError: 108519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1517.735518] LNetError: 108519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1517.749815] LNet: Removed LNI 10.1.2.3@tcp99 [ 1518.112434] Key type .llcrypt unregistered [ 1518.114623] Key type ._llcrypt unregistered [ 1520.895529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1520.899278] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1523.277967] Key type ._llcrypt registered [ 1523.281456] Key type .llcrypt registered [ 1523.399976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1523.413265] alg: No test for adler32 (adler32-zlib) [ 1524.267454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.306171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1524.323381] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1524.718517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1524.734243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1525.929646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1525.960605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1525.988117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1526.001522] LNetError: 109271:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1526.021722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1526.050064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1526.311891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1526.355829] LNet: 109291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.360137] LNetError: 109291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.364499] LNet: Removed LNI 192.168.201.45@tcp [ 1526.371116] LNet: Removed LNI 10.1.2.3@tcp [ 1526.758602] Key type .llcrypt unregistered [ 1526.761173] Key type ._llcrypt unregistered [ 1532.445446] Key type ._llcrypt registered [ 1532.448878] Key type .llcrypt registered [ 1533.219571] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:42:45 (1773103365) [ 1533.440691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1533.448921] alg: No test for adler32 (adler32-zlib) [ 1534.294141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.327137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.341108] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1534.723345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.45@tcp [ 1534.767434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg145-client@tcp [ 1538.058978] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:42:50 (1773103370) [ 1538.145976] LNet: 110822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.151494] LNetError: 110822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.160932] LNet: Removed LNI 192.168.201.45@tcp [ 1538.499911] Key type .llcrypt unregistered [ 1538.502551] Key type ._llcrypt unregistered [ 1541.287232] Key type ._llcrypt registered [ 1541.288267] Key type .llcrypt registered [ 1541.376581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1541.386458] alg: No test for adler32 (adler32-zlib) [ 1542.216792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1542.232522] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1542.248761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1545.193479] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:42:58 (1773103378) [ 1545.957540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1546.769410] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:42:59 (1773103379) [ 1547.655565] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1548.515455] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:43:01 (1773103381) [ 1548.576890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.595145] LNet: 112238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.598939] LNetError: 112238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.607630] LNet: Removed LNI 192.168.201.45@tcp [ 1548.618414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.647344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1548.659137] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1549.041366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1549.072413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1552.469384] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:43:05 (1773103385) [ 1552.512321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.525656] LNet: 112943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.529548] LNetError: 112943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.543605] LNet: Removed LNI 192.168.201.45@tcp [ 1552.561656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.609409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1552.630174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1553.033473] Key type .llcrypt unregistered [ 1553.035192] Key type ._llcrypt unregistered [ 1558.525766] Key type ._llcrypt registered [ 1558.528108] Key type .llcrypt registered [ 1559.313564] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:43:11 (1773103391) [ 1559.507946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.516828] alg: No test for adler32 (adler32-zlib) [ 1560.345329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.375520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1560.389424] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1560.786729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1560.822590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1564.353599] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:43:17 (1773103397) [ 1564.401501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.423112] LNet: 114495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.428754] LNetError: 114495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.439097] LNet: Removed LNI 192.168.201.45@tcp [ 1564.450689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.493087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1564.512423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1567.753675] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:43:20 (1773103400) [ 1567.851715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.871677] LNet: 115081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.887472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.926393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1568.127185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1568.149475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1568.187927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1568.406689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1568.431439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1568.475949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.498840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.537439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1568.737567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1568.758120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1568.792620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1568.995921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1569.016811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1572.155594] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:43:24 (1773103404) [ 1572.200545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.213663] LNet: 115709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.216881] LNet: 115709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1572.232925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.267600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1572.283078] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1572.756826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1572.769507] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1573.264285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1573.308405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1573.447492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.463045] LNetError: 115993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.467234] LNet: Removed LNI 192.168.201.45@tcp [ 1573.473603] LNet: Removed LNI 192.168.201.45@tcp2 [ 1573.490569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.525363] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1573.529897] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1573.533482] LNet: 115999:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1573.806311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1573.907830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.944152] LNet: Removed LNI 192.168.201.45@tcp [ 1574.946619] LNetError: 116036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.950535] LNetError: 116036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1574.960356] LNet: Removed LNI 192.168.201.45@tcp2 [ 1574.970993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.000325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.017215] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1575.389769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1575.402330] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1575.874479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1575.898969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1575.937418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1576.077449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.090314] LNet: 116321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.093915] LNet: 116321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1577.120126] LNet: Removed LNI 192.168.201.45@tcp [ 1577.122193] LNetError: 116321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.125588] LNetError: 116321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1577.139272] LNet: Removed LNI 192.168.201.45@tcp2 [ 1577.151346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.184600] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1577.188943] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1577.192325] LNet: 116327:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.434633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1577.559507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.576223] LNet: Removed LNI 192.168.201.45@tcp [ 1578.598578] LNet: Removed LNI 192.168.201.45@tcp2 [ 1578.611654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.647666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.666090] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1579.063296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1579.077519] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1579.572829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1579.607468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1579.718708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.768109] LNet: Removed LNI 192.168.201.45@tcp [ 1580.776380] LNet: Removed LNI 192.168.201.45@tcp2 [ 1580.789543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.826296] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1580.830435] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1580.834561] LNet: 116649:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1581.096512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1581.196173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.210131] LNet: 116686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.215542] LNet: 116686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1581.220485] LNetError: 116686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1581.224464] LNetError: 116686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1581.229244] LNet: Removed LNI 192.168.201.45@tcp [ 1581.238696] LNet: Removed LNI 192.168.201.45@tcp2 [ 1581.251346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.286397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1581.301404] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1581.732941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1581.745940] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1582.252417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1582.275822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1582.308944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1582.434601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.456731] LNet: Removed LNI 192.168.201.45@tcp [ 1584.487941] LNet: Removed LNI 192.168.201.45@tcp2 [ 1584.503261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.550248] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1584.555596] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1584.562706] LNet: 116977:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1584.839237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1584.950915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.964226] LNet: Removed LNI 192.168.201.45@tcp [ 1586.023709] LNet: Removed LNI 192.168.201.45@tcp2 [ 1586.041131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1586.078447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1586.095050] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1586.517802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1586.533306] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1587.035680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1587.058205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1587.094832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1587.306846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.384385] LNet: Removed LNI 192.168.201.45@tcp [ 1588.395414] LNet: Removed LNI 192.168.201.45@tcp2 [ 1588.410887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.456989] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1588.461199] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1588.733374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1588.899326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.920149] LNet: Removed LNI 192.168.201.45@tcp [ 1589.922151] LNetError: 117368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.926205] LNetError: 117368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1589.937371] LNet: Removed LNI 192.168.201.45@tcp2 [ 1589.950413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.990629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.008398] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1590.427649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1590.439131] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1590.933951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1590.971667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1590.991635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1591.025583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1591.222379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.235282] LNet: 117670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.239651] LNet: 117670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1592.288241] LNet: Removed LNI 192.168.201.45@tcp [ 1593.318982] LNet: Removed LNI 192.168.201.45@tcp2 [ 1593.335043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.377347] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1593.382029] LNet: Added LNI 192.168.201.45@tcp2 [8/256/0/180] [ 1593.388571] LNet: 117676:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1593.393154] LNet: 117676:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1593.655146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1597.194896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1598.080671] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:43:50 (1773103430) [ 1598.143559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.155950] LNet: Removed LNI 192.168.201.45@tcp [ 1599.207047] LNet: Removed LNI 192.168.201.45@tcp2 [ 1599.225267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.373660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.387182] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1599.773704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1599.786541] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1600.256527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.269958] LNet: Removed LNI 192.168.201.45@tcp [ 1600.278800] LNet: Removed LNI 10.1.2.3@tcp [ 1600.290956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.326122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1600.345588] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1600.349193] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1600.558605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1600.570895] LNet: Removed LNI 192.168.201.45@tcp [ 1600.578647] LNet: Removed LNI 10.1.2.3@tcp [ 1603.875578] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:43:56 (1773103436) [ 1603.925811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1603.953723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.009903] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1604.053658] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1604.055889] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1604.058899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1606.308322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1606.325393] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1606.707441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1606.720453] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1607.719573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.45@tcp [ 1607.746384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1607.781079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.45@tcp -m PUT -e local_timeout -r 1 [ 1607.816918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1607.856991] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1607.860073] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1608.001545] LNetError: 119374:0:(lib-move.c:4969:lnet_parse()) 192.168.201.45@tcp, src 192.168.201.45@tcp: Dropping PUT (error -108 looking up sender) [ 1608.928617] LNet: 119831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.931629] LNet: 119831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1608.934989] LNetError: 119831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.938931] LNetError: 119831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1608.944046] LNet: Removed LNI 192.168.201.45@tcp [ 1608.951560] LNet: Removed LNI 10.1.2.3@tcp [ 1609.143355] Key type .llcrypt unregistered [ 1609.144829] Key type ._llcrypt unregistered [ 1614.427098] Key type ._llcrypt registered [ 1614.428433] Key type .llcrypt registered [ 1615.097767] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:44:07 (1773103447) [ 1615.295428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1615.303948] alg: No test for adler32 (adler32-zlib) [ 1616.147395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1616.333603] LNet: 120682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.677847] Key type .llcrypt unregistered [ 1616.680477] Key type ._llcrypt unregistered [ 1619.549685] Key type ._llcrypt registered [ 1619.551594] Key type .llcrypt registered [ 1619.644138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1619.651857] alg: No test for adler32 (adler32-zlib) [ 1620.476456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1620.493978] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1624.170994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1624.264295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1624.585203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.145@tcp [ 1624.610238] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.145@tcp [ 1624.720903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1624.744111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1624.765348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1624.786638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1624.806600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1624.828371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1624.851342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1624.868323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1624.878370] LNet: 121131:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1624.889678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1624.908929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1624.928234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1624.951404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1624.975486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1624.998522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1625.345640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1625.466393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1625.485353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1625.503951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1625.521895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1625.541602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1625.564329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1625.585735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1625.604734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1625.616177] LNet: 121248:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1625.627941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1625.650383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1625.671079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1625.690506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1625.712106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1625.736698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1626.058808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.145@tcp [ 1626.184989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1626.206343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1626.229515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1626.249358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1626.270929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1626.293448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1626.317695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1626.337582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1626.358746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1626.381320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1626.405760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1626.428421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1626.451297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1626.472962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1626.816964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1626.937827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1626.956765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1626.978666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1626.997523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1627.018051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1627.036757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1627.061519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1627.086459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1627.099742] LNet: 121482:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1627.105402] LNet: 121482:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1627.117404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1627.138953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1627.159878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1627.181747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1627.208785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1627.233226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1627.423701] LNet: 121528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.427364] LNetError: 121528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.436613] LNet: Removed LNI 192.168.201.45@tcp [ 1627.658818] Key type .llcrypt unregistered [ 1627.660610] Key type ._llcrypt unregistered [ 1635.835739] Key type ._llcrypt registered [ 1635.837157] Key type .llcrypt registered [ 1636.664497] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:44:29 (1773103469) [ 1636.867405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1636.874365] alg: No test for adler32 (adler32-zlib) [ 1637.707251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.917633] LNet: 122414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.230545] Key type .llcrypt unregistered [ 1638.232557] Key type ._llcrypt unregistered [ 1641.437555] Key type ._llcrypt registered [ 1641.439559] Key type .llcrypt registered [ 1641.555898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1641.569443] alg: No test for adler32 (adler32-zlib) [ 1642.409778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1642.427621] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1646.282438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.145@tcp [ 1650.254072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.275067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1650.293345] LNet: Added LNI 192.168.201.45@tcp1 [8/256/0/180] [ 1650.819657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1651.110991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.145@tcp [ 1651.228928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1651.249500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1651.267249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1651.289121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1651.314817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1651.335456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1651.357081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1651.376993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1651.391090] LNet: 123064:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1651.402928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1651.421970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1651.444911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1651.465664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1651.484802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1651.504630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1651.832518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1651.949448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1651.969180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1651.986957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1652.009524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1652.037211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1652.065697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1652.086838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1652.107501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1652.120398] LNet: 123181:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1652.132429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1652.156161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1652.179466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1652.199104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1652.222215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1652.244898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1652.569330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.145@tcp [ 1652.681811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1652.703248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1652.709221] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.145@tcp [ 1652.721535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1652.743609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1652.762586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1652.785523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1652.803585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1652.825114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1652.844457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1652.868619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1652.892310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1652.917543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1652.937593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1652.958630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1653.268923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1653.398580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1653.418864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1653.439422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1653.457854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1653.477399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1653.496392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1653.515653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1653.533295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1653.544730] LNet: 123415:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1653.549017] LNet: 123415:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1653.560476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1653.581350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1653.600686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1653.622421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1653.641865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1653.661821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1656.364675] LNet: 123496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.369644] LNetError: 123496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.373866] LNet: Removed LNI 192.168.201.45@tcp [ 1656.381915] LNet: Removed LNI 192.168.201.45@tcp1 [ 1656.579862] Key type .llcrypt unregistered [ 1656.581753] Key type ._llcrypt unregistered [ 1663.910651] Key type ._llcrypt registered [ 1663.912366] Key type .llcrypt registered [ 1664.624688] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 20:44:57 (1773103497) [ 1667.974444] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 20:45:00 (1773103500) [ 1671.543339] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 20:45:04 (1773103504) [ 1674.967382] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 20:45:07 (1773103507) [ 1678.387726] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 20:45:11 (1773103511) [ 1681.917285] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 20:45:14 (1773103514) [ 1685.523865] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 20:45:18 (1773103518) [ 1689.096912] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:45:21 (1773103521) [ 1693.121376] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:45:25 (1773103525) [ 1696.796152] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:45:29 (1773103529) [ 1700.422826] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:45:33 (1773103533) [ 1704.103242] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:45:36 (1773103536) [ 1707.616431] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:45:40 (1773103540) [ 1711.022376] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:45:43 (1773103543) [ 1714.992465] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:45:47 (1773103547) [ 1718.990300] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:45:51 (1773103551) [ 1722.592600] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:45:55 (1773103555) [ 1726.186543] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:45:58 (1773103558) [ 1726.421878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1726.430378] alg: No test for adler32 (adler32-zlib) [ 1727.264361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.337317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.352786] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1728.041727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.058585] LNet: 134197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.063769] LNetError: 134197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.075675] LNet: Removed LNI 192.168.201.45@tcp [ 1728.086542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.126538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1728.147608] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1731.723530] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:46:04 (1773103564) [ 1735.256269] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:46:08 (1773103568) [ 1738.877989] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:46:11 (1773103571) [ 1742.407915] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:46:15 (1773103575) [ 1746.013342] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:46:18 (1773103578) [ 1749.690077] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:46:22 (1773103582) [ 1753.051516] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:46:25 (1773103585) [ 1756.426441] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:46:29 (1773103589) [ 1759.956316] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:46:32 (1773103592) [ 1763.461592] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:46:36 (1773103596) [ 1766.812569] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:46:39 (1773103599) [ 1769.787678] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:46:42 (1773103602) [ 1772.819778] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1518 sec ========== 20:46:45 (1773103605) [ 1772.937360] LNet: 141655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1772.940982] LNetError: 141655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1772.950594] LNet: Removed LNI 192.168.201.45@tcp [ 1773.243870] Key type .llcrypt unregistered [ 1773.245762] Key type ._llcrypt unregistered [ 1779.021349] Key type ._llcrypt registered [ 1779.023446] Key type .llcrypt registered [ 1779.308762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1779.317648] alg: No test for adler32 (adler32-zlib) [ 1780.246953] Lustre: Lustre: Build Version: 2.17.50_236_gea5aa0c [ 1780.389634] LNet: Added LNI 192.168.201.45@tcp [8/256/0/180] [ 1782.017654] Key type lgssc registered [ 1782.681536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1812.842939] Lustre: Mounted lustre-client [ 1815.920194] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1824.357360] LustreError: 144103:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff968289f7a800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1824.387037] Lustre: Unmounted lustre-client [ 1839.005374] Key type lgssc unregistered [ 1839.152350] LNet: 144586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.156983] LNetError: 144586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.169472] LNet: Removed LNI 192.168.201.45@tcp [ 1839.531490] Key type .llcrypt unregistered [ 1839.534203] Key type ._llcrypt unregistered