[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 522251694 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002268] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.006790] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.010095] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.014065] *** VALIDATE selinux *** [ 0.023631] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028645] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030079] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032124] *** VALIDATE tmpfs *** [ 0.034371] *** VALIDATE proc *** [ 0.036181] *** VALIDATE cgroup *** [ 0.037010] *** VALIDATE cgroup2 *** [ 0.038494] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039225] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041043] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045274] debug: unmapping init [mem 0xffffffff9c259000-0xffffffff9c260fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048768] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050014] ... bit width: 48 [ 0.051013] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053013] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056322] rcu: Hierarchical SRCU implementation. [ 0.058584] smp: Bringing up secondary CPUs ... [ 0.059615] x86: Booting SMP configuration: [ 0.060024] .... node #0, CPUs: #1 #2 #3 [ 0.065016] smp: Brought up 1 node, 4 CPUs [ 0.067014] smpboot: Max logical packages: 1 [ 0.068016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.170412] node 0 deferred pages initialised in 102ms [ 0.175230] devtmpfs: initialized [ 0.176345] x86/mm: Memory block size: 128MB [ 0.179482] gcov: version magic: 0x41383552 [ 0.181380] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.182148] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.183327] pinctrl core: initialized pinctrl subsystem [ 0.184294] [ 0.184963] ************************************************************* [ 0.185024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186018] ** ** [ 0.187015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.188017] ** ** [ 0.189015] ** This means that this kernel is built to expose internal ** [ 0.190014] ** IOMMU data structures, which may compromise security on ** [ 0.191013] ** your system. ** [ 0.192012] ** ** [ 0.193015] ** If you see this message and you are not debugging the ** [ 0.194014] ** kernel, report this immediately to your vendor! ** [ 0.195188] ** ** [ 0.196016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.197014] ************************************************************* [ 0.198769] NET: Registered protocol family 16 [ 0.199616] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.200061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.201065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.202470] cpuidle: using governor menu [ 0.205371] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.208495] PCI: Using configuration type 1 for base access [ 0.211137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225146] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.226024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.228073] cryptd: max_cpu_qlen set to 1000 [ 0.230454] ACPI: Added _OSI(Module Device) [ 0.231028] ACPI: Added _OSI(Processor Device) [ 0.232000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.238000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.243383] ACPI: Interpreter enabled [ 0.244051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245012] ACPI: Using IOAPIC for interrupt routing [ 0.247089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.249377] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.259000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.266258] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270552] acpiphp: Slot [2] registered [ 0.272087] acpiphp: Slot [5] registered [ 0.273073] acpiphp: Slot [6] registered [ 0.274079] acpiphp: Slot [3] registered [ 0.275069] acpiphp: Slot [4] registered [ 0.277084] acpiphp: Slot [7] registered [ 0.278068] acpiphp: Slot [8] registered [ 0.279071] acpiphp: Slot [9] registered [ 0.280074] acpiphp: Slot [10] registered [ 0.281068] acpiphp: Slot [11] registered [ 0.283070] acpiphp: Slot [12] registered [ 0.284090] acpiphp: Slot [13] registered [ 0.285066] acpiphp: Slot [14] registered [ 0.287061] acpiphp: Slot [15] registered [ 0.287999] acpiphp: Slot [16] registered [ 0.289029] acpiphp: Slot [17] registered [ 0.290081] acpiphp: Slot [18] registered [ 0.291074] acpiphp: Slot [19] registered [ 0.292095] acpiphp: Slot [20] registered [ 0.293084] acpiphp: Slot [21] registered [ 0.294070] acpiphp: Slot [22] registered [ 0.295094] acpiphp: Slot [23] registered [ 0.296082] acpiphp: Slot [24] registered [ 0.296975] acpiphp: Slot [25] registered [ 0.298096] acpiphp: Slot [26] registered [ 0.299075] acpiphp: Slot [27] registered [ 0.301068] acpiphp: Slot [28] registered [ 0.302067] acpiphp: Slot [29] registered [ 0.303050] acpiphp: Slot [30] registered [ 0.304068] acpiphp: Slot [31] registered [ 0.305047] PCI host bridge to bus 0000:00 [ 0.306013] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.308016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.309017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.311020] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.313037] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.317044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.320246] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.323424] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.327724] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.342602] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.351020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.354022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.357691] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.360972] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.363039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.366980] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.370080] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.383014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.387014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392470] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.400015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.405013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414983] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422089] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.436028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.444218] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.465020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.475349] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.477303] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.481301] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.483260] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.486644] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.492028] iommu: Default domain type: Passthrough [ 0.494417] SCSI subsystem initialized [ 0.496299] ACPI: bus type USB registered [ 0.497093] usbcore: registered new interface driver usbfs [ 0.499071] usbcore: registered new interface driver hub [ 0.500053] usbcore: registered new device driver usb [ 0.501137] pps_core: LinuxPPS API ver. 1 registered [ 0.503008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.506049] PTP clock support registered [ 0.508092] EDAC MC: Ver: 3.0.0 [ 0.509413] PCI: Using ACPI for IRQ routing [ 0.511000] NetLabel: Initializing [ 0.513014] NetLabel: domain hash size = 128 [ 0.515009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.518085] NetLabel: unlabeled traffic allowed by default [ 0.521158] vgaarb: loaded [ 0.522645] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.525015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.535000] clocksource: Switched to clocksource kvm-clock [ 0.671963] VFS: Disk quotas dquot_6.6.0 [ 0.673550] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.677232] *** VALIDATE ramfs *** [ 0.679148] *** VALIDATE hugetlbfs *** [ 0.681380] pnp: PnP ACPI init [ 0.684772] pnp: PnP ACPI: found 6 devices [ 0.707103] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.711810] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.715297] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.717291] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.722020] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.724548] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.727437] NET: Registered protocol family 2 [ 0.730662] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.738116] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.743576] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.751185] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.756511] TCP: Hash tables configured (established 65536 bind 65536) [ 0.761333] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.765512] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.769616] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.774624] NET: Registered protocol family 1 [ 0.778126] RPC: Registered named UNIX socket transport module. [ 0.780662] RPC: Registered udp transport module. [ 0.782590] RPC: Registered tcp transport module. [ 0.785403] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.788296] NET: Registered protocol family 44 [ 0.790103] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.792915] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.796190] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.800039] PCI: CLS 0 bytes, default 64 [ 0.802192] Unpacking initramfs... [ 2.523097] debug: unmapping init [mem 0xffff995a3cc64000-0xffff995a3ffcffff] [ 2.527626] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.530487] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.534300] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.204138] Initialise system trusted keyrings [ 3.206757] Key type blacklist registered [ 3.209037] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.220938] zbud: loaded [ 3.225102] *** VALIDATE nfs *** [ 3.226569] *** VALIDATE nfs4 *** [ 3.229372] pstore: using deflate compression [ 3.233175] Platform Keyring initialized [ 3.353102] NET: Registered protocol family 38 [ 3.355576] Key type asymmetric registered [ 3.356773] Asymmetric key parser 'x509' registered [ 3.358582] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.362027] io scheduler mq-deadline registered [ 3.363343] io scheduler kyber registered [ 3.364364] io scheduler bfq registered [ 3.366086] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.368531] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.370794] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.372757] ACPI: Power Button [PWRF] [ 3.376782] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.382354] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.390707] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.415580] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.446310] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.451639] Non-volatile memory driver v1.3 [ 3.453404] Linux agpgart interface v0.103 [ 3.491306] virtio_blk virtio1: [vda] 134608 512-byte logical blocks (68.9 MB/65.7 MiB) [ 3.495305] vda: detected capacity change from 0 to 68919296 [ 3.523179] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.525926] vdb: detected capacity change from 0 to 1073741824 [ 3.544402] libphy: Fixed MDIO Bus: probed [ 3.557967] usbcore: registered new interface driver usbserial_generic [ 3.560191] usbserial: USB Serial support registered for generic [ 3.561881] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.564651] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.565969] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.568121] mousedev: PS/2 mouse device common for all mice [ 3.571985] rtc_cmos 00:05: RTC can wake from S4 [ 3.572386] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.577091] rtc_cmos 00:05: registered as rtc0 [ 3.580193] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.580933] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.583437] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.586939] intel_pstate: CPU model not supported [ 3.593359] hid: raw HID events driver (C) Jiri Kosina [ 3.595065] usbcore: registered new interface driver usbhid [ 3.596186] usbhid: USB HID core driver [ 3.597394] drop_monitor: Initializing network drop monitor service [ 3.599616] Initializing XFRM netlink socket [ 3.601184] NET: Registered protocol family 10 [ 3.603349] Segment Routing with IPv6 [ 3.604295] NET: Registered protocol family 17 [ 3.606380] mpls_gso: MPLS GSO support [ 3.613266] RAS: Correctable Errors collector initialized. [ 3.614853] AVX version of gcm_enc/dec engaged. [ 3.616281] AES CTR mode by8 optimization enabled [ 3.698684] sched_clock: Marking stable (3698580303, 0)->(4848001541, -1149421238) [ 3.704582] registered taskstats version 1 [ 3.707578] Loading compiled-in X.509 certificates [ 3.710686] zswap: loaded using pool lzo/zbud [ 3.745728] Key type big_key registered [ 3.759935] Key type encrypted registered [ 3.761765] ima: No TPM chip found, activating TPM-bypass! [ 3.764466] ima: Allocated hash algorithm: sha1 [ 3.766876] ima: No architecture policies found [ 3.768953] evm: Initialising EVM extended attributes: [ 3.771721] evm: security.selinux [ 3.773414] evm: security.ima [ 3.774617] evm: security.capability [ 3.776122] evm: HMAC attrs: 0x1 [ 3.779053] rtc_cmos 00:05: setting system clock to 2026-04-29 20:23:18 UTC (1777494198) [ 3.787476] debug: unmapping init [mem 0xffffffff9d203000-0xffffffff9d3fffff] [ 3.791288] debug: unmapping init [mem 0xffffffff9bf82000-0xffffffff9c258fff] [ 3.801079] Write protecting the kernel read-only data: 28672k [ 3.805077] debug: unmapping init [mem 0xffffffff9a603000-0xffffffff9a7fffff] [ 3.808607] debug: unmapping init [mem 0xffffffff9af14000-0xffffffff9affffff] [ 3.845846] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.856125] systemd[1]: Detected virtualization kvm. [ 3.859018] systemd[1]: Detected architecture x86-64. [ 3.862233] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.891840] systemd[1]: No hostname configured. [ 3.893943] systemd[1]: Set hostname to . [ 3.896298] random: systemd: uninitialized urandom read (16 bytes read) [ 3.900095] systemd[1]: Initializing machine ID from random generator. [ 3.981546] random: ln: uninitialized urandom read (6 bytes read) [ 4.120606] random: systemd: uninitialized urandom read (16 bytes read) [ 4.123643] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.131778] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 4.137766] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.844121] device-mapper: uevent: version 1.0.3 [ 4.846631] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ 5.131351] random: fast init done [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.726252] virtio_net virtio0 ens2: renamed from eth0 [ 5.811490] scsi host0: ata_piix [ 5.842086] scsi host1: ata_piix [ 5.844123] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.847962] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.654851] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.282663] random: crng init done [ 10.285101] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.938206] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.270189] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.599805] SELinux: Disabled at runtime. [ 12.662040] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.674454] systemd[1]: Detected virtualization kvm. [ 12.676271] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.276675] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.280912] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.287589] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.293496] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.297965] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.308085] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.317130] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 13.374484] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.815140] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 14.191619] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.233077] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.354625] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.370059] EDAC sbridge: Ver: 1.1.2 [ 15.649226] Key type dns_resolver registered [ 16.013042] NFS: Registering the id_resolver key type [ 16.015282] Key type id_resolver registered [ 16.017250] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg134-client login: [ 48.884124] hrtimer: interrupt took 4087344 ns [ 83.448937] libcfs: loading out-of-tree module taints kernel. [ 83.554558] Key type ._llcrypt registered [ 83.558056] Key type .llcrypt registered [ 84.514533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 84.555508] alg: No test for adler32 (adler32-zlib) [ 86.489224] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 87.697783] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 89.631989] Key type lgssc registered [ 92.651262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 277.333465] Lustre: Mounted lustre-client [ 284.322372] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 301.160719] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing check_logdir /tmp/testlogs/ [ 303.071625] Lustre: lustre-OST0000-osc-ffff995a8522f000: disconnect after 23s idle [ 308.360852] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing yml_node [ 314.834975] Lustre: DEBUG MARKER: Client: 2.17.52.93 [ 318.758154] Lustre: DEBUG MARKER: MDS: 2.17.52.93 [ 322.622327] Lustre: DEBUG MARKER: OSS: 2.17.52.93 [ 325.140824] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Apr 29 16:28:37 EDT 2026 [ 349.446781] LustreError: 7903:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff995a8522f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 349.504682] Lustre: Unmounted lustre-client [ 387.080366] Key type lgssc unregistered [ 387.331929] LNet: 8350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.339194] LNetError: 8350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 387.351296] LNet: Removed LNI 192.168.201.34@tcp [ 388.333140] Key type .llcrypt unregistered [ 388.335458] Key type ._llcrypt unregistered [ 430.138881] Key type ._llcrypt registered [ 430.142147] Key type .llcrypt registered [ 430.299762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 430.318560] alg: No test for adler32 (adler32-zlib) [ 431.350355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 431.506129] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 431.570399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 437.432841] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 437.741848] LNet: 9232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 437.768861] LNetError: 9232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 437.810934] LNet: Removed LNI 192.168.201.34@tcp [ 438.894474] Key type .llcrypt unregistered [ 438.897848] Key type ._llcrypt unregistered [ 447.950868] Key type ._llcrypt registered [ 447.957152] Key type .llcrypt registered [ 449.495445] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:30:42 (1777494642) [ 451.058776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 451.083861] alg: No test for adler32 (adler32-zlib) [ 452.096984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.223551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.276469] LNet: 9646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.304758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.367895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.393886] LNet: 9653:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 462.012198] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:30:55 (1777494655) [ 462.104758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.151814] LNet: 10227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 462.208512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.367272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 473.541521] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:31:05 (1777494665) [ 473.732040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.855325] LNet: 10815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 473.904518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.138805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 485.315881] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:31:17 (1777494677) [ 485.484641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.526562] LNet: 11403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.558785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.710548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 497.126201] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:31:29 (1777494689) [ 497.332199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.418315] LNet: 11986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.495282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.696383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 498.767599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 506.994994] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:31:40 (1777494700) [ 507.255300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.293516] LNet: 12582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 507.328592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.415574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 507.734903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 508.357058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 517.412613] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:31:50 (1777494710) [ 517.723540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.830958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.920521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 528.091186] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:32:00 (1777494720) [ 528.336197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.394915] LNet: 13763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 528.414215] LNet: 13763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 528.460981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.586203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 528.642967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 529.384217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 529.517977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 530.234023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 530.297912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 531.072988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 531.157731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 532.020850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 532.077468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 532.914507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 532.972810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 533.840428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 533.892468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 534.634269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 534.691814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 535.301726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 535.345909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 536.163475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 536.195446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 536.821254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 536.893341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 538.139631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 538.215600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 539.348172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 539.465974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 550.608745] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:32:23 (1777494743) [ 550.774796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.884818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.067712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 551.121554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 561.426800] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:32:34 (1777494754) [ 561.888456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.930157] LNet: 15036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.940383] LNet: 15036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 562.000503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.120614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 562.177972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 571.251375] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:32:44 (1777494764) [ 571.443836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.533546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.715245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 571.751155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 581.092566] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:32:54 (1777494774) [ 581.450174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.531450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.639423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 581.675605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 592.263367] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:33:04 (1777494784) [ 592.660273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.727915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.798510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 592.880250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 603.463684] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:33:16 (1777494796) [ 603.650505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.706205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.896951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 603.971972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 613.189614] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:33:25 (1777494805) [ 613.493038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.607972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.777871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 614.468994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 624.483667] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:33:37 (1777494817) [ 624.567329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.629526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.914784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 625.229800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 634.360304] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:33:47 (1777494827) [ 634.683397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.752896] LNet: 19186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.763369] LNet: 19186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 634.810587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.996597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 635.389951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 645.526483] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:33:58 (1777494838) [ 645.846539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.947424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.944799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 647.506882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 657.653465] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:34:10 (1777494850) [ 657.882638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.980415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.128829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 658.242148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 669.082862] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:34:21 (1777494861) [ 669.236434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.376973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.693590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.788360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 680.332452] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:34:33 (1777494873) [ 680.518496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.611606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.797589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 690.464272] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:34:43 (1777494883) [ 690.600869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.698470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.832310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 700.486433] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:34:53 (1777494893) [ 701.116641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.173692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.328285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 702.150454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 711.970485] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:35:04 (1777494904) [ 712.382574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.434801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.548858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 712.659104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 722.947988] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:35:15 (1777494915) [ 723.102898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.191975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.372786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 723.400320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 732.693279] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:35:25 (1777494925) [ 732.779579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 732.859986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.087961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 733.148056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 742.313331] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:35:34 (1777494934) [ 742.576031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.744791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.953451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 743.008889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 753.715819] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:35:46 (1777494946) [ 753.926567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 754.067718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.340987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 754.490736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 754.534554] LNetError: 25778:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 754.603482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 754.665270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 763.790053] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:35:56 (1777494956) [ 763.922495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.962914] LNet: 26347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.978315] LNet: 26347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 764.009359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.118991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.yqDsn4Xu7L [ 764.234447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.yqDsn4Xu7L [ 764.341078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.yqDsn4Xu7L [ 773.397854] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:36:05 (1777494965) [ 773.561148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.650188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.821025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 773.880343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 773.926345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 783.435643] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:36:15 (1777494975) [ 783.616470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.699338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.833205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 788.282823] LNet: Message forwarding will be enabled in 2s [ 791.392445] LNet: Message forwarding enabled [ 792.002984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 792.065481] LNet: Message forwarding will stop in 2s to 6s [ 795.621199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 798.133930] LNet: Message forwarding disabled [ 798.975374] LNet: Message forwarding will be enabled in 2s [ 802.271106] LNet: Message forwarding enabled [ 808.596967] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:36:41 (1777495001) [ 809.119338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.395554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.484514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 820.076991] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:36:52 (1777495012) [ 820.230639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.317787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.463593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 820.546798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 820.624560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 820.715936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 820.789586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 820.853510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 820.959433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 821.047551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 821.122411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 821.186976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 821.258413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 821.320273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 821.421547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 821.502842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 821.596430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 821.711065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 821.787435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 821.877310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 821.934771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 822.009307] LNetError: 28844:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 830.917752] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:37:03 (1777495023) [ 831.286097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 831.381736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.462025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 841.187881] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:37:14 (1777495034) [ 843.551688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg134-server [ 846.281729] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:37:19 (1777495039) [ 846.807200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.897468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.018635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 847.086652] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 849.325098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[35-39]@tcp [ 849.397133] LNet: 30562:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 849.489924] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.36@tcp is being used as a gateway but routing feature is not turned on [ 849.523332] LNetError: 30352:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.36@tcp has gone from up to down [ 850.493887] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.35@tcp is being used as a gateway but routing feature is not turned on [ 850.519991] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 851.494845] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.36@tcp is being used as a gateway but routing feature is not turned on [ 851.531314] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 853.555509] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.35@tcp is being used as a gateway but routing feature is not turned on [ 853.576700] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 857.647360] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.36@tcp is being used as a gateway but routing feature is not turned on [ 857.681524] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 859.975972] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:37:32 (1777495052) [ 862.558667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg134-server [ 865.611293] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:37:38 (1777495058) [ 865.839964] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.35@tcp is being used as a gateway but routing feature is not turned on [ 865.878128] LNetError: 30352:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 39 previous similar messages [ 865.957917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.000530] LNetError: 31499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.051472] LNet: Removed LNI 192.168.201.34@tcp [ 867.157253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.309253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.362367] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 868.534691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[35-40/2]@tcp [ 868.613943] LNet: 31617:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 868.635693] LNet: 31617:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 868.693311] LNetError: 31503:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.35@tcp has gone from up to down [ 868.723550] LNetError: 31503:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 868.724538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[35-40/2]@tcp [ 880.064684] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:37:52 (1777495072) [ 880.228698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.260963] LNetError: 32204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.319606] LNet: Removed LNI 192.168.201.34@tcp [ 881.406032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.597343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 881.744244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.832346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.933358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 882.062168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.131439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.231943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 882.378499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.475932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.706657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 882.946403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.022130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.088759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 883.206327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.263310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.338345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 883.466165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.510323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.615466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 883.717798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.776193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.848480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 884.016248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.099443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.182847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 884.295750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.342361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.406921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 884.512978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.555992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.653387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 884.767035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.835454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.939914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 893.697902] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:38:06 (1777495086) [ 896.507512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg134-server [ 899.036926] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:38:11 (1777495091) [ 901.991907] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg134-server [ 904.621276] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:38:17 (1777495097) [ 905.914508] Key type .llcrypt unregistered [ 905.919910] Key type ._llcrypt unregistered [ 915.149045] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 915.284347] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 915.308044] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 915.319206] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 917.290593] Key type ._llcrypt registered [ 917.296112] Key type .llcrypt registered [ 917.698757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 917.748469] alg: No test for adler32 (adler32-zlib) [ 918.725665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.789704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 918.811691] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 920.035913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 920.106252] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 923.067977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 923.127611] LNet: Removed LNI 10.1.2.3@tcp [ 923.341720] LNet: 34328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.359067] LNetError: 34328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.389074] LNet: Removed LNI 192.168.201.34@tcp [ 924.296707] Key type .llcrypt unregistered [ 924.299603] Key type ._llcrypt unregistered [ 942.632739] Key type ._llcrypt registered [ 942.634490] Key type .llcrypt registered [ 944.416676] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:38:57 (1777495137) [ 945.488501] Key type .llcrypt unregistered [ 945.492932] Key type ._llcrypt unregistered [ 953.475307] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.781082] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.788809] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.808205] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 956.693490] Key type ._llcrypt registered [ 956.697587] Key type .llcrypt registered [ 956.947854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.968868] alg: No test for adler32 (adler32-zlib) [ 958.099622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.222827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 958.279837] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 959.199551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 959.242621] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 963.324780] LNet: 35820:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.330814] LNetError: 35820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.337784] LNet: Removed LNI 192.168.201.34@tcp [ 963.351383] LNet: Removed LNI 10.1.2.3@tcp [ 964.143593] Key type .llcrypt unregistered [ 964.145764] Key type ._llcrypt unregistered [ 981.661984] Key type ._llcrypt registered [ 981.668614] Key type .llcrypt registered [ 983.510945] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:39:36 (1777495176) [ 984.378631] Key type .llcrypt unregistered [ 984.388316] Key type ._llcrypt unregistered [ 993.355340] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 993.653969] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 993.672443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 993.685182] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.760669] Key type ._llcrypt registered [ 996.763898] Key type .llcrypt registered [ 997.079056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.107372] alg: No test for adler32 (adler32-zlib) [ 998.290754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.352304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 998.372653] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1001.548438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1001.579656] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1004.523311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1004.561196] LNet: Removed LNI 10.1.2.3@tcp [ 1004.581532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 1004.638544] LNet: Removed LNI 10.1.2.31@tcp [ 1004.657600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1004.698417] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1005.838768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1005.864783] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1006.806612] LNet: 37741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1006.811689] LNetError: 37741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1006.819177] LNet: Removed LNI 10.1.2.3@tcp [ 1006.831102] LNet: Removed LNI 10.1.2.31@tcp [ 1007.675945] Key type .llcrypt unregistered [ 1007.684158] Key type ._llcrypt unregistered [ 1024.097311] Key type ._llcrypt registered [ 1024.109107] Key type .llcrypt registered [ 1025.520663] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:40:18 (1777495218) [ 1026.432922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.462971] alg: No test for adler32 (adler32-zlib) [ 1027.531333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.653812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1027.687551] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1028.907459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1028.953715] LNet: 38694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.968311] LNetError: 38694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.992503] LNet: Removed LNI 192.168.201.34@tcp [ 1029.026845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.116449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.34@tcp [ 1029.141478] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1030.063544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.34@tcp [ 1030.089558] LNet: Removed LNI 192.168.201.34@tcp [ 1030.156617] LNet: 38830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.099476] Key type .llcrypt unregistered [ 1031.105150] Key type ._llcrypt unregistered [ 1047.577757] Key type ._llcrypt registered [ 1047.582801] Key type .llcrypt registered [ 1049.846828] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:40:42 (1777495242) [ 1052.846444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg134-server [ 1055.742934] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:40:48 (1777495248) [ 1056.775692] Key type .llcrypt unregistered [ 1056.782571] Key type ._llcrypt unregistered [ 1064.620117] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1064.863346] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1064.885799] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1064.904683] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1067.844858] Key type ._llcrypt registered [ 1067.850655] Key type .llcrypt registered [ 1068.439153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1068.550536] alg: No test for adler32 (adler32-zlib) [ 1069.643667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1069.801147] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 1071.517805] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1075.856135] LNet: 40665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.872700] LNetError: 40665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1075.891063] LNet: Removed LNI 192.168.201.34@tcp1 [ 1075.908304] LNet: Removed LNI 10.1.2.3@tcp2 [ 1077.275775] Key type .llcrypt unregistered [ 1077.284575] Key type ._llcrypt unregistered [ 1097.329697] Key type ._llcrypt registered [ 1097.334528] Key type .llcrypt registered [ 1099.868579] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:41:31 (1777495291) [ 1100.408830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1100.431587] alg: No test for adler32 (adler32-zlib) [ 1101.521481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.630669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1101.696202] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1103.193795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1103.239898] LNet: 41635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1103.262990] LNetError: 41635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1103.285329] LNet: Removed LNI 192.168.201.34@tcp [ 1103.331973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1103.468387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1103.513522] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1113.913814] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:41:46 (1777495306) [ 1114.356424] LNet: 42227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.365938] LNetError: 42227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.397252] LNet: Removed LNI 192.168.201.34@tcp [ 1115.404945] Key type .llcrypt unregistered [ 1115.418184] Key type ._llcrypt unregistered [ 1124.238534] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1124.531803] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1124.545964] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1124.602307] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1127.523715] Key type ._llcrypt registered [ 1127.532713] Key type .llcrypt registered [ 1127.879209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1127.922531] alg: No test for adler32 (adler32-zlib) [ 1128.977453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1129.044415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1129.103876] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1130.244205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1130.301910] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1134.284477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1134.387495] LNet: 42972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.410788] LNetError: 42972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.418108] LNet: Removed LNI 192.168.201.34@tcp [ 1134.447362] LNet: Removed LNI 10.1.2.3@tcp [ 1134.522221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1134.677367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1134.723484] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1134.730714] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1135.219705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1135.246366] LNet: 42999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.253688] LNetError: 42999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.260671] LNetError: 42999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1135.266286] LNet: Removed LNI 192.168.201.34@tcp [ 1135.276848] LNet: Removed LNI 10.1.2.3@tcp [ 1135.301865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.372632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1135.398783] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1135.407985] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1135.896834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1135.935511] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 1137.746850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1137.787540] LNet: 43161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.797715] LNetError: 43161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1137.805714] LNetError: 43161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1137.820040] LNet: Removed LNI 192.168.201.34@tcp [ 1137.824603] LNet: Removed LNI 10.1.2.3@tcp [ 1137.840809] LNet: Removed LNI 192.168.201.34@tcp2 [ 1137.882042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1138.026713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1138.097087] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1138.106276] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1138.124121] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 1149.029496] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:42:21 (1777495341) [ 1149.324197] LNet: 43759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.341383] LNetError: 43759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.356866] LNetError: 43759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1149.367153] LNet: Removed LNI 192.168.201.34@tcp [ 1149.374532] LNet: Removed LNI 10.1.2.3@tcp [ 1149.397471] LNet: Removed LNI 192.168.201.34@tcp2 [ 1150.136769] Key type .llcrypt unregistered [ 1150.141469] Key type ._llcrypt unregistered [ 1158.429590] Key type ._llcrypt registered [ 1158.431665] Key type .llcrypt registered [ 1158.618829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.633719] alg: No test for adler32 (adler32-zlib) [ 1159.785387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1159.856635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1159.882685] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1161.138614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1161.175646] LNet: 44155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.195571] LNetError: 44155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1161.214070] LNet: Removed LNI 192.168.201.34@tcp [ 1161.240616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1161.378157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1161.416172] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1162.079556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1162.113691] LNet: 44185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.125445] LNetError: 44185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.169190] LNet: Removed LNI 192.168.201.34@tcp [ 1162.202668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1162.289333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1162.318145] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1162.997713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1163.074274] LNet: Removed LNI 192.168.201.34@tcp [ 1163.115544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.196584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1163.236974] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1163.783108] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1163.940336] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1163.957322] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1163.977869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1166.609492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1166.662408] LNet: 44294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1166.673371] LNet: 44294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1166.684682] LNetError: 44294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.694457] LNetError: 44294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1166.726667] LNet: Removed LNI 192.168.201.34@tcp [ 1166.761964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1166.883609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1166.952870] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1177.451260] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:42:50 (1777495370) [ 1177.681241] LNet: 44886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.695656] LNetError: 44886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.726429] LNet: Removed LNI 192.168.201.34@tcp [ 1178.791082] Key type .llcrypt unregistered [ 1178.792965] Key type ._llcrypt unregistered [ 1187.645895] Key type ._llcrypt registered [ 1187.648616] Key type .llcrypt registered [ 1188.081618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1188.125486] alg: No test for adler32 (adler32-zlib) [ 1189.194082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1189.276544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1189.327968] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1190.540195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1190.606475] LNet: 45294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.639580] LNetError: 45294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.688224] LNet: Removed LNI 192.168.201.34@tcp [ 1190.747744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1190.863239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1190.908458] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1200.598812] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:43:13 (1777495393) [ 1200.788844] LNet: 45879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.797855] LNetError: 45879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1200.845422] LNet: Removed LNI 192.168.201.34@tcp [ 1201.996687] Key type .llcrypt unregistered [ 1202.002255] Key type ._llcrypt unregistered [ 1212.074368] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1212.433077] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1212.457782] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1212.479104] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1215.394811] Key type ._llcrypt registered [ 1215.401627] Key type .llcrypt registered [ 1215.750737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1215.767625] alg: No test for adler32 (adler32-zlib) [ 1216.932680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1217.042965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1217.086437] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1218.766652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1218.838233] LNet: 46338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.864544] LNetError: 46338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.917183] LNet: Removed LNI 192.168.201.34@tcp [ 1218.998686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1219.118189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1219.196942] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1229.836120] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:43:42 (1777495422) [ 1230.164067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1230.197259] LNet: 46937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.204657] LNetError: 46937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.248456] LNet: Removed LNI 192.168.201.34@tcp [ 1230.278105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1230.387818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1238.719201] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:43:51 (1777495431) [ 1239.143812] LNet: 47519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.150872] Key type .llcrypt unregistered [ 1240.154892] Key type ._llcrypt unregistered [ 1249.535266] Key type ._llcrypt registered [ 1249.539921] Key type .llcrypt registered [ 1249.834908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1249.864952] alg: No test for adler32 (adler32-zlib) [ 1250.879681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1250.956569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1250.997859] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1252.547468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1252.594723] LNet: 47919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1252.601504] LNetError: 47919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.616703] LNet: Removed LNI 192.168.201.34@tcp [ 1252.642398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1252.765394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1252.807812] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1262.781566] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:44:15 (1777495455) [ 1263.189329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1263.291687] LNet: 48514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.315619] LNetError: 48514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.336651] LNet: Removed LNI 192.168.201.34@tcp [ 1263.401122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1263.542557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1272.364237] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:44:24 (1777495464) [ 1274.520128] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1277.623263] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:44:30 (1777495470) [ 1280.173229] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1283.200289] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:44:35 (1777495475) [ 1286.817622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1290.178189] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:44:42 (1777495482) [ 1292.824488] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1296.233448] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:44:48 (1777495488) [ 1298.673618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1301.302568] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:44:54 (1777495494) [ 1301.664884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1301.699561] LNet: 50570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.738033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1301.880697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1301.944938] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1303.142558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.34 [ 1303.181523] LNet: 50703:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1303.209185] LNet: Removed LNI 192.168.201.34@tcp [ 1312.500706] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:45:05 (1777495505) [ 1312.857321] LNet: 51275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.871680] LNetError: 51275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1312.890871] LNet: Removed LNI 192.168.201.34@tcp [ 1313.610865] Key type .llcrypt unregistered [ 1313.615496] Key type ._llcrypt unregistered [ 1323.163528] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1323.524625] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1323.539598] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1323.551142] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1326.495549] Key type ._llcrypt registered [ 1326.498452] Key type .llcrypt registered [ 1326.723331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1326.772520] alg: No test for adler32 (adler32-zlib) [ 1328.652614] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1337.083401] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:45:30 (1777495530) [ 1337.217873] LNet: 52172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.234774] LNetError: 52172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1337.252829] LNet: Removed LNI 10.1.2.3@tcp [ 1337.989463] Key type .llcrypt unregistered [ 1337.992317] Key type ._llcrypt unregistered [ 1346.914044] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1347.217726] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1347.235424] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1347.249774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1349.242435] Key type ._llcrypt registered [ 1349.254171] Key type .llcrypt registered [ 1349.694326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1349.737783] alg: No test for adler32 (adler32-zlib) [ 1350.856850] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1361.063435] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:45:53 (1777495553) [ 1361.305867] LNet: 53059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.333575] LNetError: 53059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1361.390201] LNet: Removed LNI 10.1.2.3@tcp [ 1362.445672] Key type .llcrypt unregistered [ 1362.450986] Key type ._llcrypt unregistered [ 1375.319424] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1375.740915] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1375.750975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1375.757215] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1379.598706] Key type ._llcrypt registered [ 1379.604299] Key type .llcrypt registered [ 1380.025422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1380.088587] alg: No test for adler32 (adler32-zlib) [ 1381.361794] LNet: 53367:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1381.395980] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1392.785187] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:46:24 (1777495584) [ 1393.003842] LNet: 53955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.019869] LNetError: 53955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1393.067158] LNet: Removed LNI 10.1.2.3@tcp [ 1394.058706] Key type .llcrypt unregistered [ 1394.060547] Key type ._llcrypt unregistered [ 1406.140615] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1406.604923] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1406.613653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1406.619958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1408.704706] Key type ._llcrypt registered [ 1408.717760] Key type .llcrypt registered [ 1409.247746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1409.292605] alg: No test for adler32 (adler32-zlib) [ 1410.391917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.569839] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1425.683703] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:46:58 (1777495618) [ 1426.282193] LNet: 55183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.293392] LNetError: 55183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1426.310066] LNet: Removed LNI 10.1.2.3@tcp [ 1427.297221] Key type .llcrypt unregistered [ 1427.299945] Key type ._llcrypt unregistered [ 1436.520535] Key type ._llcrypt registered [ 1436.526588] Key type .llcrypt registered [ 1436.807693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1436.841524] alg: No test for adler32 (adler32-zlib) [ 1437.969480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1438.023417] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1450.607880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 1451.713561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_interrupt -r 1 [ 1451.818680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_interrupt -r 1 [ 1451.922988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1451.975409] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1452.384098] LNet: 1 local NIs in recovery (showing 1): 192.168.201.34@tcp [ 1452.472868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1452.561044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1452.637896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1452.712579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1452.812793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1452.885809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1452.957884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1453.029294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1453.057310] LNet: 55621:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1453.080603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1453.163765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1453.217882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.269595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1453.333349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1453.416352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1453.519322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1454.773272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_dropped -r 1 [ 1454.868920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_dropped -r 1 [ 1454.972986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1455.042633] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1455.054258] LNet: Skipped 1 previous similar message [ 1455.449427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1455.498742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1455.538840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1455.572391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1455.613595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1455.664975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1455.759066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1455.819276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1455.873296] LNet: 55730:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1455.977876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1456.059814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1456.146725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1456.210938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1456.272877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1456.336304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1456.417192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1457.524288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_aborted -r 1 [ 1457.565768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_aborted -r 1 [ 1457.626995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1457.660317] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1457.993059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1458.062854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1458.144819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1458.228939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1458.306399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1458.381824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1458.472662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1458.536778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1458.577481] LNet: 55840:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1458.617344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1458.711907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1458.782270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1458.848783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1458.921790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1459.006782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1459.112470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1460.283421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_no_route -r 1 [ 1460.340738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_no_route -r 1 [ 1460.385658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1460.402637] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1460.681409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1460.743852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1460.794190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1460.897710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1460.963809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1461.040636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1461.140227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1461.233870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1461.280069] LNet: 55949:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1461.330270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1461.402914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1461.471696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1461.624947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1461.657470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1461.691451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1461.719925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1462.361637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_timeout -r 1 [ 1462.417662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_timeout -r 1 [ 1462.457944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1462.740926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1462.782130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1462.828367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1462.874606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1462.917991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1462.990956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1463.050412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1463.115136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1463.223578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1463.253440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1463.304090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1463.344922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1463.396306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1463.487991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1463.552941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1464.779518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_error -r 1 [ 1464.858866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_error -r 1 [ 1464.901316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1464.928751] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1464.936812] LNet: Skipped 2 previous similar messages [ 1465.387741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1465.475280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1465.547959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1465.637331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1465.722971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1465.803656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1465.896648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1465.973723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1466.063643] LNet: 56168:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1466.078924] LNet: 56168:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1466.121812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1466.234149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1466.369246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1466.470763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1466.565933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1466.619814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1466.661801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1467.172988] LNet: 56209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.186860] LNetError: 56209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.223988] LNet: Removed LNI 192.168.201.34@tcp [ 1468.157365] Key type .llcrypt unregistered [ 1468.166608] Key type ._llcrypt unregistered [ 1494.098273] Key type ._llcrypt registered [ 1494.105790] Key type .llcrypt registered [ 1496.032738] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:48:08 (1777495688) [ 1497.267716] Key type .llcrypt unregistered [ 1497.271711] Key type ._llcrypt unregistered [ 1505.054294] Key type ._llcrypt registered [ 1505.057956] Key type .llcrypt registered [ 1505.553580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1505.596990] alg: No test for adler32 (adler32-zlib) [ 1506.606812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1506.650307] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1518.033914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 1535.933986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.028804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1536.100744] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 1537.708793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_interrupt -r 1 [ 1537.767755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_interrupt -r 1 [ 1537.829299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_interrupt -r 1 [ 1537.868578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_interrupt -r 1 [ 1537.916662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_interrupt -r 1 [ 1537.957961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_interrupt -r 1 [ 1538.000861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_interrupt -r 1 [ 1538.026660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_interrupt -r 1 [ 1538.056430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1538.074137] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1538.078884] LNet: 1 local NIs in recovery (showing 1): 192.168.201.34@tcp [ 1538.284816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1538.308611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1538.332918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1538.355748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1538.382633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1538.406622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1538.444402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1538.486624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1538.535839] LNet: 57679:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1538.558230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1538.608183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1538.634372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1538.664084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1538.695615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1538.743454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1538.801425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1539.448907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_dropped -r 1 [ 1539.480790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_dropped -r 1 [ 1539.525787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_dropped -r 1 [ 1539.561932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_dropped -r 1 [ 1539.591704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_dropped -r 1 [ 1539.622792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_dropped -r 1 [ 1539.655500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_dropped -r 1 [ 1539.687905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_dropped -r 1 [ 1539.715769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1539.736265] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1539.740439] LNet: Skipped 2 previous similar messages [ 1539.955991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1539.980403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1540.009293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1540.036279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1540.066755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1540.090889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1540.113568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1540.142429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1540.157588] LNet: 57801:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1540.175055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1540.195787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1540.241623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1540.343996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1540.389965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1540.440384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1540.466885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1540.931891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_aborted -r 1 [ 1541.022760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_aborted -r 1 [ 1541.140854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_aborted -r 1 [ 1541.182921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_aborted -r 1 [ 1541.256046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_aborted -r 1 [ 1541.285918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_aborted -r 1 [ 1541.323669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_aborted -r 1 [ 1541.371803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_aborted -r 1 [ 1541.433980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1541.465622] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1541.477151] LNet: Skipped 2 previous similar messages [ 1541.804912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1541.871222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1541.912711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1541.936991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1541.979689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1542.058347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1542.100079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1542.145297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1542.169229] LNet: 57922:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1542.192318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1542.223790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1542.251142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1542.306056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1542.336825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1542.369932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1542.432735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1543.161291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_no_route -r 1 [ 1543.219824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_no_route -r 1 [ 1543.268832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_no_route -r 1 [ 1543.357991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_no_route -r 1 [ 1543.437837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_no_route -r 1 [ 1543.516736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_no_route -r 1 [ 1543.602779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_no_route -r 1 [ 1543.708632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_no_route -r 1 [ 1543.791518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1543.830152] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1543.836913] LNet: Skipped 2 previous similar messages [ 1544.238718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1544.272790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1544.310614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1544.348743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1544.391620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1544.430806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1544.472213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1544.535441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1544.562502] LNet: 58043:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1544.594677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1544.694430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1544.765565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1544.791753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1544.828990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1544.875771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1544.914763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1545.591694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_timeout -r 1 [ 1545.639776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_timeout -r 1 [ 1545.712540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_timeout -r 1 [ 1545.788729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_timeout -r 1 [ 1545.852766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_timeout -r 1 [ 1545.900904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_timeout -r 1 [ 1545.977629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_timeout -r 1 [ 1546.044340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_timeout -r 1 [ 1546.100274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1546.488722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1546.529936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1546.568638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1546.615739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1546.666164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1546.702954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1546.734440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1546.767874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1546.809797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1546.847670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1546.883389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1546.915889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1546.950570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1546.995784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1547.044398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1547.851975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_error -r 1 [ 1547.910403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_error -r 1 [ 1547.976585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_error -r 1 [ 1548.035931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_error -r 1 [ 1548.068611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_error -r 1 [ 1548.133722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_error -r 1 [ 1548.214972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_error -r 1 [ 1548.290272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_error -r 1 [ 1548.413731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1548.490792] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1548.512441] LNet: Skipped 7 previous similar messages [ 1548.907757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1548.950708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1548.985346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1549.034978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1549.068498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1549.109682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1549.139768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1549.192704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1549.222891] LNet: 58286:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1549.232420] LNet: 58286:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1549.247712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1549.302371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1549.366676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1549.417946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1549.476484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1549.551357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1549.624569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1557.289744] LNet: 58362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.322374] LNetError: 58362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.372560] LNet: Removed LNI 192.168.201.34@tcp [ 1558.390367] LNetError: 58362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.413633] LNet: Removed LNI 192.168.201.34@tcp1 [ 1559.348792] Key type .llcrypt unregistered [ 1559.358727] Key type ._llcrypt unregistered [ 1586.031894] Key type ._llcrypt registered [ 1586.036116] Key type .llcrypt registered [ 1589.301651] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:49:40 (1777495780) [ 1591.105095] Key type .llcrypt unregistered [ 1591.120574] Key type ._llcrypt unregistered [ 1602.356969] Key type ._llcrypt registered [ 1602.364182] Key type .llcrypt registered [ 1602.621371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1602.680530] alg: No test for adler32 (adler32-zlib) [ 1603.792356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1603.837024] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1615.949565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 1616.835467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e remote_dropped -r 1 [ 1616.874755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e remote_dropped -r 1 [ 1616.977365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1617.030781] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1617.389395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1617.448873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1617.515249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1617.565328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1617.617709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1617.671746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.715717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.761334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1617.786345] LNet: 59615:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1617.808595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1617.857433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1617.890068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1617.930407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1617.988145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.072974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1618.172140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1619.189752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e remote_error -r 1 [ 1619.236843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e remote_error -r 1 [ 1619.281078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1619.304760] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1619.599539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1619.631620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1619.665956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1619.699821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1619.744989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1619.785940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1619.819419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1619.868370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1619.899228] LNet: 59724:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1619.935803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1619.996820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1620.074108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1620.134147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1620.233871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1620.306660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1620.393647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1621.115658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e remote_timeout -r 1 [ 1621.165590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e remote_timeout -r 1 [ 1621.225728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1621.273356] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1621.694619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1621.763720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1621.823403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1621.863855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1621.898693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1621.958628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1622.007587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1622.065525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1622.081769] LNet: 59834:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1622.110535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1622.146656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1622.201662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1622.260340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1622.308891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1622.350844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1622.400655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1622.795726] LNet: 59875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.809724] LNetError: 59875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.843964] LNet: Removed LNI 192.168.201.34@tcp [ 1623.550602] Key type .llcrypt unregistered [ 1623.556816] Key type ._llcrypt unregistered [ 1650.586440] Key type ._llcrypt registered [ 1650.589404] Key type .llcrypt registered [ 1652.430342] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:50:45 (1777495845) [ 1653.603532] Key type .llcrypt unregistered [ 1653.607490] Key type ._llcrypt unregistered [ 1663.515329] Key type ._llcrypt registered [ 1663.519205] Key type .llcrypt registered [ 1663.865879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1663.892417] alg: No test for adler32 (adler32-zlib) [ 1664.989923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1665.076740] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1675.214886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 1687.252044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.294138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1687.362384] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 1689.642767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e remote_dropped -r 1 [ 1689.671815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e remote_dropped -r 1 [ 1689.709363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e remote_dropped -r 1 [ 1689.752252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e remote_dropped -r 1 [ 1689.799297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e remote_dropped -r 1 [ 1689.851513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e remote_dropped -r 1 [ 1689.890782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e remote_dropped -r 1 [ 1689.922730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e remote_dropped -r 1 [ 1689.957706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1689.994794] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1690.002866] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.134@tcp [ 1690.297386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1690.339252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1690.372706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1690.405256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1690.421720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1690.447771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1690.519484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1690.582492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1690.645859] LNet: 61344:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1690.674426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1690.727919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1690.773886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1690.806702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1690.847385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1690.876450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1690.903494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1690.926099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1691.718717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e remote_error -r 1 [ 1691.771984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e remote_error -r 1 [ 1691.871246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e remote_error -r 1 [ 1691.965077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e remote_error -r 1 [ 1692.071529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e remote_error -r 1 [ 1692.139557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e remote_error -r 1 [ 1692.209344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e remote_error -r 1 [ 1692.260731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e remote_error -r 1 [ 1692.336762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1692.367398] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1692.372730] LNet: Skipped 2 previous similar messages [ 1692.812403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1692.883424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1692.949977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1693.010477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1693.080512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1693.182256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1693.237714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1693.285200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1693.322355] LNet: 61468:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1693.353317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1693.393876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1693.461196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1693.535395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1693.633973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1693.711946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1693.792432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1693.893042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1694.594821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e remote_timeout -r 1 [ 1694.637344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e remote_timeout -r 1 [ 1694.680293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e remote_timeout -r 1 [ 1694.741981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e remote_timeout -r 1 [ 1694.809554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e remote_timeout -r 1 [ 1694.914554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e remote_timeout -r 1 [ 1694.967328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e remote_timeout -r 1 [ 1695.007431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e remote_timeout -r 1 [ 1695.049474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.134@tcp [ 1695.069604] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1695.076225] LNet: Skipped 1 previous similar message [ 1695.386880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1695.435472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1695.483591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1695.538893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1695.589356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1695.655241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1695.724874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1695.770979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1695.790393] LNet: 61591:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1695.807979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1695.850746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1695.890970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1695.929625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1696.000464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1696.040259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1696.110513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1696.183932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1704.209550] LNet: 61669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.225748] LNetError: 61669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.243653] LNet: Removed LNI 192.168.201.34@tcp [ 1704.279811] LNet: Removed LNI 192.168.201.34@tcp1 [ 1705.129064] Key type .llcrypt unregistered [ 1705.135846] Key type ._llcrypt unregistered [ 1728.837705] Key type ._llcrypt registered [ 1728.841035] Key type .llcrypt registered [ 1730.593401] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:52:03 (1777495923) [ 1731.620771] Key type .llcrypt unregistered [ 1731.624630] Key type ._llcrypt unregistered [ 1741.257048] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1741.523357] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1741.543675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1741.561450] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1743.447258] Key type ._llcrypt registered [ 1743.457940] Key type .llcrypt registered [ 1743.863808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1743.898677] alg: No test for adler32 (adler32-zlib) [ 1744.884881] LNet: 62744:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1744.902654] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1744.959486] LNet: 62758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.970823] LNetError: 62758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.992811] LNet: Removed LNI 192.168.201.34@tcp [ 1746.189687] Key type .llcrypt unregistered [ 1746.204823] Key type ._llcrypt unregistered [ 1755.564780] Key type ._llcrypt registered [ 1755.569445] Key type .llcrypt registered [ 1755.883548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1755.905758] alg: No test for adler32 (adler32-zlib) [ 1756.928330] LNet: 63017:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1756.944661] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1756.947332] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1756.989476] LNet: 63031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1756.996455] LNetError: 63031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.003693] LNet: Removed LNI 192.168.201.34@tcp [ 1757.018887] LNet: Removed LNI 10.1.2.3@tcp1 [ 1758.044599] Key type .llcrypt unregistered [ 1758.055240] Key type ._llcrypt unregistered [ 1768.893674] Key type ._llcrypt registered [ 1768.895948] Key type .llcrypt registered [ 1769.294330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1769.337191] alg: No test for adler32 (adler32-zlib) [ 1770.545641] LNet: 63291:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1770.585439] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1770.590914] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1770.662257] LNet: 63305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1770.679280] LNetError: 63305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1770.688288] LNet: Removed LNI 192.168.201.34@tcp [ 1770.698310] LNet: Removed LNI 10.1.2.3@tcp [ 1771.920736] Key type .llcrypt unregistered [ 1771.922990] Key type ._llcrypt unregistered [ 1781.676560] Key type ._llcrypt registered [ 1781.680377] Key type .llcrypt registered [ 1782.013187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1782.047469] alg: No test for adler32 (adler32-zlib) [ 1783.120524] LNet: 63566:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1783.152363] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1783.158818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1783.262661] LNet: 63580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.275704] LNetError: 63580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.292748] LNet: Removed LNI 192.168.201.34@tcp [ 1783.316320] LNet: Removed LNI 10.1.2.3@tcp [ 1784.386306] Key type .llcrypt unregistered [ 1784.394603] Key type ._llcrypt unregistered [ 1794.852688] Key type ._llcrypt registered [ 1794.861874] Key type .llcrypt registered [ 1795.119622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1795.166034] alg: No test for adler32 (adler32-zlib) [ 1796.272531] LNet: 63845:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1796.299370] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1796.303935] LNetError: 63845:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1796.311038] LNetError: Error -100 starting up LNI tcp [ 1796.332255] LNet: Removed LNI 192.168.201.34@tcp [ 1797.320582] Key type .llcrypt unregistered [ 1797.324276] Key type ._llcrypt unregistered [ 1814.587587] Key type ._llcrypt registered [ 1814.594403] Key type .llcrypt registered [ 1816.654351] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:53:28 (1777496008) [ 1818.190424] Key type .llcrypt unregistered [ 1818.195115] Key type ._llcrypt unregistered [ 1827.822056] Key type ._llcrypt registered [ 1827.830982] Key type .llcrypt registered [ 1828.294501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1828.329107] alg: No test for adler32 (adler32-zlib) [ 1829.308027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1829.365180] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1840.792926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 1841.913483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e network_timeout -r 1 [ 1842.025041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e network_timeout -r 1 [ 1842.127254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 1842.216387] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1842.650643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1842.712562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1842.738871] LNet: 1 local NIs in recovery (showing 1): 192.168.201.34@tcp [ 1842.805875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1842.883715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1843.009031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1843.094398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1843.169997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1843.240590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1843.270357] LNet: 65103:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1843.319147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1843.396781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1843.452228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1843.566526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1843.699687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1843.773630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1843.820399] LNet: There was an unexpected network error while writing to 192.168.201.34: rc = -22 [ 1844.250324] LNet: 65142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.268446] LNetError: 65142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.305292] LNet: Removed LNI 192.168.201.34@tcp [ 1845.154366] Key type .llcrypt unregistered [ 1845.159743] Key type ._llcrypt unregistered [ 1874.037596] Key type ._llcrypt registered [ 1874.050868] Key type .llcrypt registered [ 1874.525532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1874.581922] alg: No test for adler32 (adler32-zlib) [ 1875.570479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1875.605737] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1886.806076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 1902.502795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.575602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1902.624780] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 1904.741643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e network_timeout -r 1 [ 1904.814087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e network_timeout -r 1 [ 1904.862486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e network_timeout -r 1 [ 1904.921902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e network_timeout -r 1 [ 1904.988382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e network_timeout -r 1 [ 1905.056557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e network_timeout -r 1 [ 1905.128588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e network_timeout -r 1 [ 1905.229527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e network_timeout -r 1 [ 1905.333926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 1905.373430] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1905.748083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1905.815616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1905.858492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1905.905798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1905.949992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1905.980290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1906.021139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1906.073535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1906.107700] LNet: 66024:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1906.148479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1906.200504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1906.260913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1906.325333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1906.338774] LNet: 1 local NIs in recovery (showing 1): 192.168.201.34@tcp [ 1906.339387] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1906.350659] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.134@tcp [ 1906.378220] LNet: Skipped 1 previous similar message [ 1906.402578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1906.468432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1913.097851] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 1913.662439] LNet: 66098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.685042] LNetError: 66098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.694469] LNet: Removed LNI 192.168.201.34@tcp [ 1913.715474] LNet: Removed LNI 192.168.201.34@tcp1 [ 1914.516406] Key type .llcrypt unregistered [ 1914.518453] Key type ._llcrypt unregistered [ 1940.656933] Key type ._llcrypt registered [ 1940.663911] Key type .llcrypt registered [ 1942.768600] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:55:35 (1777496135) [ 1943.599616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1943.618404] alg: No test for adler32 (adler32-zlib) [ 1944.726651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1944.827296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1944.859829] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1946.133521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1946.157619] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 1947.737738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 1947.890376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1948.002792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1948.080562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1948.140437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1948.210715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1948.296945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.34@tcp [ 1948.347567] LNetError: 67200:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.34@tcp to recovery [ 1948.900409] LNet: 1 local NIs in recovery (showing 1): 192.168.201.34@tcp [ 1948.903701] LNet: There was an unexpected network error while writing to 192.168.201.34: rc = -22 [ 1951.008941] LNet: There was an unexpected network error while writing to 192.168.201.34: rc = -22 [ 1952.267979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1955.103640] LNet: There was an unexpected network error while writing to 192.168.201.34: rc = -22 [ 1956.235030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1956.280173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1956.387382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1956.456519] LNet: 67307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.477474] LNetError: 67307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.505702] LNet: Removed LNI 192.168.201.34@tcp [ 1956.523134] LNet: Removed LNI 192.168.201.34@tcp1 [ 1956.591818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.659519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1956.688802] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1958.140611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1958.162933] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 1959.828525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 1959.900781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1959.994865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1960.066742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1960.111858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1960.157644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1960.220781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1960.317720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.34@tcp [ 1960.337584] LNetError: 67580:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.34@tcp to recovery [ 1960.735418] LNet: There was an unexpected network error while writing to 192.168.201.34: rc = -22 [ 1964.073476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1966.882063] LNet: There was an unexpected network error while writing to 192.168.201.34: rc = -22 [ 1966.891724] LNet: Skipped 1 previous similar message [ 1971.494214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1971.577900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1971.722102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1984.319411] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:56:16 (1777496176) [ 1984.470437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1984.536821] LNet: 68283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.558862] LNetError: 68283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1984.567990] LNetError: 68283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1984.584888] LNet: Removed LNI 192.168.201.34@tcp [ 1984.607058] LNet: Removed LNI 192.168.201.34@tcp1 [ 1984.681797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.791969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1984.813777] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 1986.282520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1986.341208] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 1988.216909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 1988.391669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1988.455311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1988.502019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1988.613598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1988.697183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1988.791992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.34@tcp --health 0 [ 1989.003163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1989.857139] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.34@tcp [ 1989.858227] LNet: There was an unexpected network error while writing to 192.168.201.34: rc = -22 [ 1989.895210] LNet: Skipped 1 previous similar message [ 2003.249556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2003.548894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2003.602899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2003.663515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.34@tcp --health 500 [ 2003.802443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2009.699619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2009.811278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2009.839441] LNet: 68756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.847716] LNetError: 68756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.854469] LNetError: 68756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2009.872794] LNet: Removed LNI 192.168.201.34@tcp [ 2009.898365] LNet: Removed LNI 192.168.201.34@tcp1 [ 2009.956751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.149966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2010.208496] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2011.502575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2011.534506] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 2012.806685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 2012.897381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2012.994401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 2013.044719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 2013.112141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 2013.156952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2013.210269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 2013.256883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.34@tcp --health 0 [ 2014.117105] LNet: There was an unexpected network error while writing to 192.168.201.34: rc = -22 [ 2014.140369] LNet: Skipped 3 previous similar messages [ 2014.770744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2025.684583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2025.774221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2025.827552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2035.689736] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:57:08 (1777496228) [ 2036.392777] LNet: 69802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.415497] LNetError: 69802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.431165] LNetError: 69802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2036.451971] LNet: Removed LNI 192.168.201.34@tcp [ 2036.483051] LNet: Removed LNI 192.168.201.34@tcp1 [ 2037.181931] Key type .llcrypt unregistered [ 2037.185934] Key type ._llcrypt unregistered [ 2046.231668] Key type ._llcrypt registered [ 2046.238645] Key type .llcrypt registered [ 2046.484350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2046.531709] alg: No test for adler32 (adler32-zlib) [ 2047.734643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2047.822819] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2063.429923] Lustre: DEBUG MARKER: Initial discovery [ 2063.520693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.134@tcp [ 2068.083535] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 2068.193968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 2068.296502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.134@tcp [ 2068.355461] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 2068.437923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2068.567789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.134@tcp -r 1 -m GET -l 3 [ 2068.635643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.134@tcp -r 1 -m GET -e local_error [ 2068.721771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.134@tcp -r 1 -m PUT -l 6 [ 2069.343645] LNet: 1 local NIs in recovery (showing 1): 192.168.201.34@tcp [ 2071.366746] Lustre: DEBUG MARKER: Force oleg134-client.virtnet to discover 192.168.201.134@tcp (in background) [ 2071.436507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.134@tcp [ 2073.801802] Lustre: DEBUG MARKER: Force oleg134-server to discover 192.168.201.34@tcp [ 2074.527971] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 2077.593179] Lustre: DEBUG MARKER: Wait for 70558 [ 2077.855777] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 2079.580853] Lustre: DEBUG MARKER: Finished wait on 70558 [ 2079.924117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 2080.002586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 2080.437851] LNet: 71019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2080.449165] LNetError: 71019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2081.524446] LNet: Removed LNI 192.168.201.34@tcp [ 2082.041745] Key type .llcrypt unregistered [ 2082.044734] Key type ._llcrypt unregistered [ 2106.834392] Key type ._llcrypt registered [ 2106.839249] Key type .llcrypt registered [ 2109.162419] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:58:21 (1777496301) [ 2110.166567] Key type .llcrypt unregistered [ 2110.168898] Key type ._llcrypt unregistered [ 2120.515867] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2120.775071] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2120.793526] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2120.804924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2122.805597] Key type ._llcrypt registered [ 2122.813221] Key type .llcrypt registered [ 2123.111630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2123.144396] alg: No test for adler32 (adler32-zlib) [ 2124.344870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.527220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2124.666268] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2126.256981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2126.303474] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2142.251129] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:58:54 (1777496334) [ 2142.725484] LNet: 73092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.752592] LNetError: 73092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2142.770480] LNet: Removed LNI 192.168.201.34@tcp [ 2142.804369] LNet: Removed LNI 10.1.2.3@tcp [ 2143.900480] Key type .llcrypt unregistered [ 2143.905163] Key type ._llcrypt unregistered [ 2155.401626] Key type ._llcrypt registered [ 2155.404435] Key type .llcrypt registered [ 2155.876819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2155.928087] alg: No test for adler32 (adler32-zlib) [ 2156.943543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.051992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2157.128301] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2158.634948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2158.686464] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2160.625873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.34@tcp [ 2161.759161] LNetError: Unexpected error -22 connecting to 192.168.201.34@tcp at host 192.168.201.34:988 [ 2170.518891] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:59:23 (1777496363) [ 2170.718452] LNet: 74215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.733775] LNetError: 74215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.743132] LNet: Removed LNI 192.168.201.34@tcp [ 2170.781547] LNet: Removed LNI 10.1.2.3@tcp [ 2171.812638] Key type .llcrypt unregistered [ 2171.821409] Key type ._llcrypt unregistered [ 2181.617446] Key type ._llcrypt registered [ 2181.630566] Key type .llcrypt registered [ 2182.046392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2182.079855] alg: No test for adler32 (adler32-zlib) [ 2183.132366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.272751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2183.355586] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 2184.548937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2184.591443] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 2186.312424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.34@tcp1 --nid 192.168.201.34@tcp2 [ 2193.052566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.34@tcp1 192.168.201.34@tcp2 [ 2193.091959] LNetError: 75060:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.34@tcp1 to 192.168.201.34@tcp2 - Route table may be misconfigured [ 2193.128285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.34@tcp2 192.168.201.34@tcp1 [ 2204.369372] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:59:56 (1777496396) [ 2204.535118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2204.607804] LNet: 75627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.620361] LNetError: 75627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.635569] LNet: Removed LNI 192.168.201.34@tcp1 [ 2205.673063] LNet: Removed LNI 192.168.201.34@tcp2 [ 2205.775269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.908188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.970794] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2207.270228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2207.307834] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 2209.355704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 2209.476325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.34@tcp 192.168.201.34@tcp [ 2209.558148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.34@tcp1 192.168.201.34@tcp1 [ 2209.637191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.34@tcp -d 192.168.201.34@tcp -e network_timeout [ 2209.689451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -e network_timeout [ 2209.782320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -e network_timeout [ 2209.828828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -e network_timeout [ 2209.907450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2209.940294] LNet: There was an unexpected network error while writing to 192.168.201.34: rc = -22 [ 2209.983131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 2210.163222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 2210.380682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 2210.989024] LNet: 1 local NIs in recovery (showing 1): 192.168.201.34@tcp [ 2221.029172] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:00:13 (1777496413) [ 2221.194834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.272428] LNet: 76490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.293962] LNetError: 76490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.312698] LNetError: 76490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2221.327307] LNet: Removed LNI 192.168.201.34@tcp [ 2222.375114] LNet: Removed LNI 192.168.201.34@tcp1 [ 2222.457563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.654559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 2223.220514] LNet: 76534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.793797] Key type .llcrypt unregistered [ 2223.802539] Key type ._llcrypt unregistered [ 2242.422206] Key type ._llcrypt registered [ 2242.427199] Key type .llcrypt registered [ 2244.724211] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:00:36 (1777496436) [ 2245.455479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2245.487155] alg: No test for adler32 (adler32-zlib) [ 2246.494864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.852207] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 2251.459836] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:00:44 (1777496444) [ 2251.764105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.809595] LNet: 77633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.865961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.980568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2252.050118] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2253.355468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2253.403536] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 2254.924988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2254.974306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp1 [ 2255.025841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp1 [ 2264.752217] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:00:56 (1777496456) [ 2268.317108] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg134-server [ 2270.645278] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:01:03 (1777496463) [ 2273.560267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg134-server [ 2276.536372] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:01:08 (1777496468) [ 2279.812624] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg134-server [ 2281.983779] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:01:14 (1777496474) [ 2284.291422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg134-server [ 2286.835155] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:01:19 (1777496479) [ 2289.890757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg134-server [ 2292.741764] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:01:25 (1777496485) [ 2295.489047] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg134-server [ 2298.166358] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:01:30 (1777496490) [ 2300.963528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg134-server [ 2303.858534] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:01:36 (1777496496) [ 2306.704129] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg134-server [ 2309.272660] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:01:41 (1777496501) [ 2312.088135] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg134-server [ 2314.926613] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:01:47 (1777496507) [ 2315.365538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2315.422815] LNet: 81529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.458368] LNetError: 81529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.475641] LNet: Removed LNI 192.168.201.34@tcp [ 2316.519918] LNet: Removed LNI 192.168.201.34@tcp1 [ 2316.579620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.722973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2316.804044] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2317.983028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2318.245339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2318.398254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2318.431243] LNet: 81676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.443307] LNetError: 81676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.450583] LNetError: 81676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2318.468506] LNet: Removed LNI 192.168.201.34@tcp [ 2318.487444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2318.538665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2318.572615] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2319.993531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2320.324993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2320.377564] LNetError: 81822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.405336] LNet: Removed LNI 192.168.201.34@tcp [ 2320.461352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.603713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2320.647458] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2321.759145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2321.864988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2322.200132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2322.270439] LNet: 81969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.282825] LNet: 81969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2323.368440] LNet: Removed LNI 192.168.201.34@tcp [ 2323.461878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.614772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2323.677508] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2325.228198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2325.437501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2325.457972] LNetError: 82114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.467725] LNetError: 82114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2326.506347] LNet: Removed LNI 192.168.201.34@tcp [ 2326.573808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.754378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.821254] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2327.929989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2328.087210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2328.391585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2328.440873] LNet: 82262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.448051] LNet: 82262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2329.510673] LNet: Removed LNI 192.168.201.34@tcp [ 2329.544461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.666607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2329.713588] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2330.969858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2331.276250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.320508] LNetError: 82407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.340236] LNetError: 82407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2331.358667] LNet: Removed LNI 192.168.201.34@tcp [ 2331.391642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.490498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2331.538231] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2332.919069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2333.023907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2333.266660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.377277] LNet: Removed LNI 192.168.201.34@tcp [ 2334.438145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.512151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2334.541224] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2335.577724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2337.055113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2337.090469] LNet: 82710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.096563] LNet: 82710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2338.152254] LNet: Removed LNI 192.168.201.34@tcp [ 2338.230149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.431423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2338.550729] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2339.836020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2339.983463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2341.597209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.694071] LNetError: 82866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2341.712121] LNetError: 82866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2342.760065] LNet: Removed LNI 192.168.201.34@tcp [ 2342.837525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.988373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2343.061956] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2344.801939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2346.395114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.438798] LNet: Removed LNI 192.168.201.34@tcp [ 2347.478258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.546785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2347.582014] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2348.616701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2348.754176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2349.174300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2350.251083] LNet: Removed LNI 192.168.201.34@tcp [ 2350.326694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.431278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2350.466826] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2351.744529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2352.216877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2353.349562] LNet: Removed LNI 192.168.201.34@tcp [ 2353.410604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.527348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2353.584687] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2354.759451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2354.843931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.856977] LNet: 83447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2354.861991] LNet: 83447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2354.877712] LNet: Removed LNI 192.168.201.34@tcp [ 2354.912975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.984131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2355.012988] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2356.743585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2368.762123] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:02:41 (1777496561) [ 2369.349148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.409459] LNetError: 84153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.424527] LNetError: 84153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2369.453586] LNet: Removed LNI 192.168.201.34@tcp [ 2369.522303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.642868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2369.675219] LNet: Added LNI 192.168.201.34@tcp231 [8/256/0/180] [ 2372.277734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2372.349063] LNet: Removed LNI 192.168.201.34@tcp231 [ 2372.442738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2372.797727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2372.849235] LNet: Added LNI 192.168.201.34@tcp231 [8/256/0/180] [ 2372.865611] LNet: 84299:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2373.690107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2373.741959] LNet: Removed LNI 192.168.201.34@tcp231 [ 2373.759276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2373.808577] LNet: Added LNI 192.168.201.34@tcp231 [8/256/0/180] [ 2375.650618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2375.717209] LNet: Removed LNI 192.168.201.34@tcp231 [ 2375.772362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.973748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2376.069100] LNet: Added LNI 192.168.201.34@tcp231 [8/256/0/0] [ 2376.086592] LNet: 84452:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2376.977398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2377.144213] LNet: Removed LNI 192.168.201.34@tcp231 [ 2377.173176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2377.209443] LNet: Added LNI 192.168.201.34@tcp231 [8/256/0/0] [ 2378.808274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2378.858260] LNet: Removed LNI 192.168.201.34@tcp231 [ 2378.900478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2379.087799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2379.219192] LNet: Added LNI 192.168.201.34@tcp231 [8/256/0/60] [ 2379.230696] LNet: 84604:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2380.440137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2380.539765] LNet: Removed LNI 192.168.201.34@tcp231 [ 2380.569566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2380.690393] LNet: Added LNI 192.168.201.34@tcp231 [8/256/0/60] [ 2383.605621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2383.632689] LNet: Removed LNI 192.168.201.34@tcp231 [ 2383.689399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.826227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2383.888404] LNet: Added LNI 192.168.201.34@tcp231 [8/256/0/180] [ 2383.926040] LNet: 84756:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2385.153787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2385.217180] LNet: Removed LNI 192.168.201.34@tcp231 [ 2385.252127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2385.304102] LNet: Added LNI 192.168.201.34@tcp231 [8/256/0/180] [ 2397.672448] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:03:10 (1777496590) [ 2398.192359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2398.267763] LNet: 85467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.284304] LNet: 85467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2398.341476] LNet: Removed LNI 192.168.201.34@tcp231 [ 2398.402262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.540320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2398.624866] LNet: Added LNI 192.168.201.34@tcp232 [8/256/0/180] [ 2399.823658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2399.882964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2399.942684] LNet: Removed LNI 192.168.201.34@tcp232 [ 2399.974223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2400.035349] LNet: Added LNI 192.168.201.34@tcp232 [8/256/0/180] [ 2401.363271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2401.423614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2401.462788] LNet: Removed LNI 192.168.201.34@tcp232 [ 2401.481672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2401.512687] LNet: Added LNI 192.168.201.34@tcp232 [8/256/0/180] [ 2402.764283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2402.833900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2402.866600] LNet: Removed LNI 192.168.201.34@tcp232 [ 2402.919709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2402.990084] LNet: Added LNI 192.168.201.34@tcp232 [8/256/0/180] [ 2404.188312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2404.299370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2404.352984] LNet: Removed LNI 192.168.201.34@tcp232 [ 2404.376729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2413.559593] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:03:26 (1777496606) [ 2414.856394] Key type .llcrypt unregistered [ 2414.859964] Key type ._llcrypt unregistered [ 2423.482625] Key type ._llcrypt registered [ 2423.492532] Key type .llcrypt registered [ 2423.767424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2423.799201] alg: No test for adler32 (adler32-zlib) [ 2424.956724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2425.093155] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2436.458083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 2451.807212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.855224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2451.875815] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 2453.587334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_interrupt -r 2 [ 2453.644258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_interrupt -r 2 [ 2453.698142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_interrupt -r 2 [ 2453.755644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_interrupt -r 2 [ 2453.827367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_interrupt -r 2 [ 2453.862742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_interrupt -r 2 [ 2453.897343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_interrupt -r 2 [ 2453.961000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_interrupt -r 2 [ 2454.057323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.34@tcp 192.168.201.134@tcp [ 2454.100276] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 2454.107808] LNet: 1 local NIs in recovery (showing 1): 192.168.201.34@tcp [ 2454.551800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2454.629373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2454.665773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2454.730421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2454.825792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2454.906252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2454.987353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2455.050955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2455.104833] LNet: 87262:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2455.177983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2455.257601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2455.349845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2455.402091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2455.470275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2455.577114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2456.328610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_dropped -r 2 [ 2456.399636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_dropped -r 2 [ 2456.486152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_dropped -r 2 [ 2456.614515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_dropped -r 2 [ 2456.685802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_dropped -r 2 [ 2456.761444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_dropped -r 2 [ 2456.811788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_dropped -r 2 [ 2456.880590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_dropped -r 2 [ 2456.927712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.34@tcp 192.168.201.134@tcp [ 2456.956600] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 2457.331230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2457.377098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2457.436972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2457.509870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2457.607139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2457.671952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2457.794519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2457.868593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2457.899906] LNet: 87383:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2457.930760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2457.998658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2458.097878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2458.145380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2458.214026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2458.287295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2459.211821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_aborted -r 2 [ 2459.248939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_aborted -r 2 [ 2459.289384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_aborted -r 2 [ 2459.334786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_aborted -r 2 [ 2459.373374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_aborted -r 2 [ 2459.419102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_aborted -r 2 [ 2459.461341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_aborted -r 2 [ 2459.502454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_aborted -r 2 [ 2459.532618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.34@tcp 192.168.201.134@tcp [ 2459.550801] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 2459.932670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2460.035693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2460.138918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2460.218073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2460.265483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2460.325341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2460.379186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2460.441202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2460.470222] LNet: 87503:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2460.497944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2460.554112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2460.622948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2460.695469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2460.808279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2460.888689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2461.798289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_no_route -r 2 [ 2461.873169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_no_route -r 2 [ 2461.927987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_no_route -r 2 [ 2462.014508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_no_route -r 2 [ 2462.106932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_no_route -r 2 [ 2462.205650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_no_route -r 2 [ 2462.268432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_no_route -r 2 [ 2462.344876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_no_route -r 2 [ 2462.407875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.34@tcp 192.168.201.134@tcp [ 2462.467994] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 2462.844869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2462.877645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2462.908630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2462.946936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2462.985612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2463.025815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2463.055976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2463.119668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2463.154720] LNet: 87624:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2463.198459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2463.271303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2463.344931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2463.430995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2463.515658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2463.575507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2464.421195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e local_timeout -r 2 [ 2464.538751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e local_timeout -r 2 [ 2464.625098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e local_timeout -r 2 [ 2464.725153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e local_timeout -r 2 [ 2464.792763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e local_timeout -r 2 [ 2464.909053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e local_timeout -r 2 [ 2465.003362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e local_timeout -r 2 [ 2465.067664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e local_timeout -r 2 [ 2465.116777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.34@tcp 192.168.201.134@tcp [ 2465.645517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2465.706441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2465.787924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2465.843701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2465.907641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2465.961688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2466.050437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2466.143480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2466.223106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2466.291731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2466.378105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2466.455678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2466.503819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2466.581477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2467.546939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp -m GET -e remote_dropped -r 2 [ 2467.651598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp -d 192.168.201.134@tcp1 -m GET -e remote_dropped -r 2 [ 2467.717545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp -m GET -e remote_dropped -r 2 [ 2467.791689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp -d 192.168.201.34@tcp1 -m GET -e remote_dropped -r 2 [ 2467.869925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp -m GET -e remote_dropped -r 2 [ 2467.962827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.34@tcp1 -d 192.168.201.134@tcp1 -m GET -e remote_dropped -r 2 [ 2468.029696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp -m GET -e remote_dropped -r 2 [ 2468.092753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.34@tcp1 -d 192.168.201.34@tcp1 -m GET -e remote_dropped -r 2 [ 2468.194912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.34@tcp 192.168.201.134@tcp [ 2468.229336] LNet: There was an unexpected network error while writing to 192.168.201.134: rc = -22 [ 2468.241623] LNet: Skipped 1 previous similar message [ 2468.253262] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.134@tcp [ 2468.585493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2468.661773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2468.713743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2468.763375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2468.819383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2468.871550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2468.913373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2468.966995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2469.009882] LNet: 87863:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2469.040332] LNet: 87863:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2469.086724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2469.189387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2469.289646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2469.376925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2469.453155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2469.530962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2477.892656] LNet: 87939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2477.906659] LNetError: 87939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2477.938312] LNet: Removed LNI 192.168.201.34@tcp [ 2477.968080] LNet: Removed LNI 192.168.201.34@tcp1 [ 2478.513078] Key type .llcrypt unregistered [ 2478.517896] Key type ._llcrypt unregistered [ 2506.559261] Key type ._llcrypt registered [ 2506.567926] Key type .llcrypt registered [ 2508.235369] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:05:00 (1777496700) [ 2508.848141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2508.869369] alg: No test for adler32 (adler32-zlib) [ 2510.005487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2510.166272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2510.246075] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2511.897647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2511.931840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.34@tcp [ 2511.958374] LNetError: 88919:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.34@tcp to recovery [ 2512.052966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.34@tcp [ 2512.283176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2521.809321] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:05:14 (1777496714) [ 2521.944484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2521.997592] LNet: 89503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2522.006953] LNetError: 89503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2522.044727] LNet: Removed LNI 192.168.201.34@tcp [ 2522.113946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.189312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2522.220176] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2523.512916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2523.542472] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 2525.128764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.34@tcp [ 2525.159807] LNetError: 89774:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.34@tcp to recovery [ 2525.226145] LNet: 1 local NIs in recovery (showing 1): 192.168.201.34@tcp [ 2525.235750] LNet: There was an unexpected network error while writing to 192.168.201.34: rc = -22 [ 2525.394021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.34@tcp [ 2537.858718] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:05:30 (1777496730) [ 2541.536117] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg134-server [ 2544.395268] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:05:36 (1777496736) [ 2547.597436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg134-server [ 2550.451627] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:05:42 (1777496742) [ 2552.980039] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg134-server [ 2555.914257] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:05:48 (1777496748) [ 2556.142464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2556.198363] LNet: 91394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2556.206957] LNetError: 91394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2556.228767] LNet: Removed LNI 192.168.201.34@tcp [ 2556.247147] LNet: Removed LNI 192.168.201.34@tcp1 [ 2556.363343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2556.502632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2556.528231] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2557.666453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2557.778360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2557.865265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2557.982789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2558.102208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2558.189330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2568.327766] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:06:00 (1777496760) [ 2568.480469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2568.515251] LNet: 92117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.530220] LNetError: 92117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2568.540963] LNetError: 92117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2568.570531] LNet: Removed LNI 192.168.201.34@tcp [ 2568.628064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2568.765553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2568.797801] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2580.313077] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:06:12 (1777496772) [ 2580.900861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2580.958262] LNet: 92829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2580.974579] LNetError: 92829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2581.020332] LNet: Removed LNI 192.168.201.34@tcp [ 2581.080829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2581.170956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2581.255081] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2592.518556] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:06:24 (1777496784) [ 2594.703342] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2597.899401] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:06:30 (1777496790) [ 2598.745404] LNet: 93866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2598.764462] LNetError: 93866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2598.779085] LNet: Removed LNI 192.168.201.34@tcp [ 2599.958713] Key type .llcrypt unregistered [ 2599.969367] Key type ._llcrypt unregistered [ 2610.762307] Key type ._llcrypt registered [ 2610.773288] Key type .llcrypt registered [ 2611.187861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2611.244515] alg: No test for adler32 (adler32-zlib) [ 2612.307867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2612.341095] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2623.730787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 2632.348675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.134@tcp [ 2632.979138] LNet: 94286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.988627] LNetError: 94286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2633.015793] LNet: Removed LNI 192.168.201.34@tcp [ 2633.905717] Key type .llcrypt unregistered [ 2633.910074] Key type ._llcrypt unregistered [ 2651.303885] Key type ._llcrypt registered [ 2651.315232] Key type .llcrypt registered [ 2653.406651] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2655.581653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2657.932191] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:07:30 (1777496850) [ 2658.770468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2658.794413] alg: No test for adler32 (adler32-zlib) [ 2659.850467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2660.065627] LNet: 95375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2661.079461] Key type .llcrypt unregistered [ 2661.095849] Key type ._llcrypt unregistered [ 2671.889329] Key type ._llcrypt registered [ 2671.895104] Key type .llcrypt registered [ 2672.224751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2672.261359] alg: No test for adler32 (adler32-zlib) [ 2673.222977] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2673.230382] LNet: 95632:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 2673.238188] LNet: 95632:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2673.313196] LNetError: 95644:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.37@tcp is being used as a gateway but routing feature is not turned on [ 2673.347985] LNetError: 95644:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.201.37@tcp has gone from up to down [ 2674.001863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2674.041894] LNet: 95690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2674.068824] LNetError: 95690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2674.098885] LNet: Removed LNI 192.168.201.34@tcp [ 2674.129503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2674.251793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2674.283185] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2674.302030] LNet: 95697:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2674.319843] LNet: 95697:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 2674.329273] LNetError: 95697:0:(peer.c:367:lnet_peer_ni_del_locked()) Peer NI 192.168.201.35@tcp is a gateway. Can not delete it [ 2674.343272] LNetError: 95697:0:(peer.c:2167:lnet_del_peer_ni()) 192.168.201.35@tcp is a router. Can not be deleted [ 2674.362213] LNet: 95697:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2674.373334] LNetError: 95694:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.36@tcp is being used as a gateway but routing feature is not turned on [ 2674.424355] LNetError: 95694:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 2674.439080] LNetError: 95694:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.201.36@tcp has gone from up to down [ 2674.455129] LNetError: 95694:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 2685.068266] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:07:57 (1777496877) [ 2688.188505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg134-server [ 2690.514726] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:08:03 (1777496883) [ 2693.355745] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg134-server [ 2696.096080] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:08:08 (1777496888) [ 2698.332726] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 2700.432780] Key type lgssc registered [ 2702.573859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2730.009582] Key type lgssc unregistered [ 2730.472962] LNet: 98123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2730.482646] LNetError: 98123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2730.505713] LNet: Removed LNI 192.168.201.34@tcp [ 2731.437160] Key type .llcrypt unregistered [ 2731.447438] Key type ._llcrypt unregistered [ 2753.871657] Key type ._llcrypt registered [ 2753.873700] Key type .llcrypt registered [ 2755.667421] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:09:08 (1777496948) [ 2756.343471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2756.374412] alg: No test for adler32 (adler32-zlib) [ 2757.494088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2757.621387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2757.689946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2757.777690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2757.849760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2766.555910] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:09:19 (1777496959) [ 2766.962439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2767.002849] LNet: 99534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2767.038902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2767.158699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2767.236883] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2768.737304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2768.782333] LNet: 99678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2768.799452] LNetError: 99678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2768.850111] LNet: Removed LNI 192.168.201.34@tcp [ 2768.884721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2768.948567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2768.978952] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2769.797060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2769.854658] LNet: 99707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2769.870184] LNetError: 99707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2769.892183] LNet: Removed LNI 192.168.201.34@tcp [ 2769.918491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2769.982533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2770.007366] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2780.616641] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:09:33 (1777496973) [ 2781.097306] LNet: 100298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2781.107433] LNetError: 100298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2781.126060] LNet: Removed LNI 192.168.201.34@tcp [ 2782.048858] Key type .llcrypt unregistered [ 2782.054493] Key type ._llcrypt unregistered [ 2792.412707] Key type ._llcrypt registered [ 2792.415784] Key type .llcrypt registered [ 2792.806743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2792.884386] alg: No test for adler32 (adler32-zlib) [ 2793.827022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2793.961293] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2793.965852] LNet: *** cfs_fail_loc=e003, val=100*** [ 2795.398531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2795.420644] LNet: *** cfs_fail_loc=e003, val=100*** [ 2795.424769] LNet: Skipped 2 previous similar messages [ 2795.435463] Ping reply received with mapping: [ 2795.438107] found metadata reply NI[192.168.201.34@tcp] [ 2795.440900] 22 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2795.444329] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2795.446857] 5a 5a 5a 5a 5a 5a 5a 5a [ 2795.644510] LNet: 100696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2795.664125] LNetError: 100696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2795.689394] LNet: Removed LNI 192.168.201.34@tcp [ 2796.675430] Key type .llcrypt unregistered [ 2796.683543] Key type ._llcrypt unregistered [ 2815.233423] Key type ._llcrypt registered [ 2815.240367] Key type .llcrypt registered [ 2817.370210] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:10:09 (1777497009) [ 2818.168278] Key type .llcrypt unregistered [ 2818.171256] Key type ._llcrypt unregistered [ 2828.474781] Key type ._llcrypt registered [ 2828.477758] Key type .llcrypt registered [ 2828.707627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2828.728748] alg: No test for adler32 (adler32-zlib) [ 2830.199497] LNetError: 101743:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2830.843312] Key type .llcrypt unregistered [ 2830.847497] Key type ._llcrypt unregistered [ 2839.259992] Key type ._llcrypt registered [ 2839.266241] Key type .llcrypt registered [ 2841.479626] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:10:33 (1777497033) [ 2842.659669] Key type .llcrypt unregistered [ 2842.664836] Key type ._llcrypt unregistered [ 2852.334268] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2852.360671] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2855.321797] Key type ._llcrypt registered [ 2855.330408] Key type .llcrypt registered [ 2855.739581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2855.774932] alg: No test for adler32 (adler32-zlib) [ 2856.853761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2856.946167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2870.770515] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:11:03 (1777497063) [ 2871.229151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2871.266666] LNet: 103525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2871.280301] LNetError: 103525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2871.301043] LNet: Removed LNI 10.1.2.3@tcp [ 2871.364431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2872.971996] Key type .llcrypt unregistered [ 2872.974146] Key type ._llcrypt unregistered [ 2882.927705] Key type ._llcrypt registered [ 2882.940315] Key type .llcrypt registered [ 2883.536669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2883.577619] alg: No test for adler32 (adler32-zlib) [ 2884.687979] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2884.780183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2885.004882] LNet: 103814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2885.013475] LNetError: 103814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2886.061273] LNet: Removed LNI 192.168.201.34@tcp [ 2887.298613] Key type .llcrypt unregistered [ 2887.304264] Key type ._llcrypt unregistered [ 2904.953901] Key type ._llcrypt registered [ 2904.956886] Key type .llcrypt registered [ 2906.946308] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:11:39 (1777497099) [ 2908.316283] Key type .llcrypt unregistered [ 2908.319994] Key type ._llcrypt unregistered [ 2920.523520] Key type ._llcrypt registered [ 2920.533531] Key type .llcrypt registered [ 2920.981263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2921.020382] alg: No test for adler32 (adler32-zlib) [ 2932.388154] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:12:04 (1777497124) [ 2932.654072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2932.816054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2933.079639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2933.149628] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2934.594878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2934.654925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2934.691967] LNet: Removed LNI 192.168.201.34@tcp [ 2935.033034] LNet: 105567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2935.518895] Key type .llcrypt unregistered [ 2935.522296] Key type ._llcrypt unregistered [ 2951.823717] Key type ._llcrypt registered [ 2951.828554] Key type .llcrypt registered [ 2954.188714] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:12:26 (1777497146) [ 2954.872651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2954.902569] alg: No test for adler32 (adler32-zlib) [ 2955.989614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2956.079255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2956.124401] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2957.298182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 2957.384436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.34@tcp [ 2957.428701] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2957.436090] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2957.447121] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2957, 0, 0 [ 2957.459565] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2957.465547] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2957.475084] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2957.480408] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2957, 0, 0 [ 2957.489173] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2957.500955] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2957.510174] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2957.514668] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2957, 0, 3007 [ 2957.537550] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2957.543598] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2957.548817] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2957.558938] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2957, 0, 3007 [ 2957.575339] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2967.059419] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:12:39 (1777497159) [ 2967.844773] LNet: 107108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2967.853568] LNetError: 107108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2967.875324] LNet: Removed LNI 192.168.201.34@tcp [ 2968.833933] Key type .llcrypt unregistered [ 2968.842551] Key type ._llcrypt unregistered [ 2979.456778] Key type ._llcrypt registered [ 2979.459181] Key type .llcrypt registered [ 2979.794924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2979.812869] alg: No test for adler32 (adler32-zlib) [ 2980.924246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2981.024592] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 2992.890919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 3008.099293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3008.186814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3008.238507] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 3009.711537] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3010.002833] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3010.006420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3010.014402] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3012.493967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 3012.533641] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 3029.143860] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:13:41 (1777497221) [ 3029.301104] LNet: 108608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3029.318442] LNetError: 108608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3029.331647] LNet: Removed LNI 192.168.201.34@tcp [ 3030.367353] LNet: Removed LNI 192.168.201.34@tcp1 [ 3030.369987] LNetError: 108608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 3030.388042] LNetError: 108608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 3030.409328] LNet: Removed LNI 10.1.2.3@tcp99 [ 3031.541933] Key type .llcrypt unregistered [ 3031.547946] Key type ._llcrypt unregistered [ 3041.260766] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3041.266647] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3044.180904] Key type ._llcrypt registered [ 3044.183721] Key type .llcrypt registered [ 3044.520443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3044.549507] alg: No test for adler32 (adler32-zlib) [ 3045.545506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3045.649950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3045.733895] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3046.777503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3046.804451] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3050.147626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 3050.210063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3050.261154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 3050.282872] LNetError: 109362:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 3050.332477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3050.390930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3051.234923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 3051.408847] LNet: 109382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3051.435453] LNetError: 109382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3051.451827] LNet: Removed LNI 192.168.201.34@tcp [ 3051.478469] LNet: Removed LNI 10.1.2.3@tcp [ 3052.621961] Key type .llcrypt unregistered [ 3052.627145] Key type ._llcrypt unregistered [ 3073.581992] Key type ._llcrypt registered [ 3073.585966] Key type .llcrypt registered [ 3075.794581] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:14:28 (1777497268) [ 3076.395493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3076.420688] alg: No test for adler32 (adler32-zlib) [ 3077.581616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3077.660442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3077.695580] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3079.005686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.34@tcp [ 3079.202902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg134-client@tcp [ 3088.827621] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:14:41 (1777497281) [ 3088.942191] LNet: 110909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3088.956658] LNetError: 110909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3088.987447] LNet: Removed LNI 192.168.201.34@tcp [ 3090.219418] Key type .llcrypt unregistered [ 3090.235960] Key type ._llcrypt unregistered [ 3099.493660] Key type ._llcrypt registered [ 3099.497308] Key type .llcrypt registered [ 3099.640705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3099.654736] alg: No test for adler32 (adler32-zlib) [ 3100.603976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 3100.647442] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3100.700798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 3109.538784] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:15:02 (1777497302) [ 3112.268910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 3115.320641] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:15:07 (1777497307) [ 3117.430532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 3121.334910] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:15:13 (1777497313) [ 3121.468840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3121.531546] LNet: 112332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3121.538794] LNetError: 112332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3121.555652] LNet: Removed LNI 192.168.201.34@tcp [ 3121.614200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3121.785142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3121.830702] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3123.065966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3123.138198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 3133.841947] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:15:26 (1777497326) [ 3133.965623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3134.006260] LNet: 113042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3134.023269] LNetError: 113042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3134.058416] LNet: Removed LNI 192.168.201.34@tcp [ 3134.157637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3134.261287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 3134.321137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3135.237610] Key type .llcrypt unregistered [ 3135.239579] Key type ._llcrypt unregistered [ 3154.818791] Key type ._llcrypt registered [ 3154.824485] Key type .llcrypt registered [ 3157.129982] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:15:49 (1777497349) [ 3157.761621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3157.785453] alg: No test for adler32 (adler32-zlib) [ 3158.907146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3158.992247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3159.030061] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3160.232590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3160.311849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3168.927398] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:16:02 (1777497362) [ 3169.145121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3169.198900] LNet: 114607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3169.214757] LNetError: 114607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3169.242621] LNet: Removed LNI 192.168.201.34@tcp [ 3169.306601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3169.369433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 3169.450328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 3179.061756] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:16:11 (1777497371) [ 3179.251731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3179.342481] LNet: 115182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3179.391807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3179.642970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3180.587594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3180.651718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3180.800779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 3181.862088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3181.942153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3182.170898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3182.229700] LNet: 115221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3182.266161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3182.406400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3183.316536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3183.385658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3183.485223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 3184.483160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 3184.571967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 3195.089661] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:16:27 (1777497387) [ 3195.338494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3195.402728] LNet: 115819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3195.460496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3195.602342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3195.674251] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3196.993553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3197.017538] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3198.496291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3198.611320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3199.077487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3199.112923] LNetError: 116102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3200.159434] LNet: Removed LNI 192.168.201.34@tcp [ 3200.161883] LNetError: 116102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3200.188642] LNet: Removed LNI 192.168.201.34@tcp2 [ 3200.294394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3200.500675] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3200.523578] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3200.533761] LNet: 116108:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3201.804410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3202.165800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3202.223239] LNet: 116145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3202.235346] LNet: 116145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 3202.250342] LNetError: 116145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3202.259886] LNet: Removed LNI 192.168.201.34@tcp [ 3202.274433] LNet: Removed LNI 192.168.201.34@tcp2 [ 3202.320191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3202.473531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3202.566867] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3203.924675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3203.956729] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3205.322435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3205.432859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3205.566834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3206.058630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3207.136519] LNet: Removed LNI 192.168.201.34@tcp [ 3207.144594] LNetError: 116429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3207.177504] LNetError: 116429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 3207.205156] LNet: Removed LNI 192.168.201.34@tcp2 [ 3207.256323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3207.368707] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3207.377474] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3207.383673] LNet: 116436:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3208.438680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3208.970816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3209.024979] LNet: Removed LNI 192.168.201.34@tcp [ 3210.089687] LNet: Removed LNI 192.168.201.34@tcp2 [ 3210.200769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3210.387899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3210.468058] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3211.750499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3211.829898] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3213.148651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3213.283611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3213.674266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3213.730509] LNet: 116753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3213.746801] LNet: 116753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 3214.818188] LNet: Removed LNI 192.168.201.34@tcp [ 3214.853332] LNet: Removed LNI 192.168.201.34@tcp2 [ 3214.968816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3215.131411] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3215.139879] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3215.148390] LNet: 116759:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3216.399817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3216.754497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3216.810632] LNetError: 116796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3216.819925] LNetError: 116796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 3216.828761] LNet: Removed LNI 192.168.201.34@tcp [ 3216.848890] LNet: Removed LNI 192.168.201.34@tcp2 [ 3216.885981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3216.973540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3217.040771] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3218.667621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3218.708980] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3220.482213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3220.604250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3220.771872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3221.179453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3222.239335] LNet: Removed LNI 192.168.201.34@tcp [ 3223.282207] LNet: Removed LNI 192.168.201.34@tcp2 [ 3223.377945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3223.636094] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3223.652635] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3223.668768] LNet: 117087:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3224.914723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3225.372488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3225.466759] LNet: Removed LNI 192.168.201.34@tcp [ 3226.537315] LNet: Removed LNI 192.168.201.34@tcp2 [ 3226.594751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3226.689731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3226.714138] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3227.786523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3227.827056] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3229.370780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 3229.457415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 3229.550915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3230.170469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3230.216374] LNet: 117421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3230.233988] LNet: 117421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 3231.263644] LNet: Removed LNI 192.168.201.34@tcp [ 3231.287076] LNet: Removed LNI 192.168.201.34@tcp2 [ 3231.368470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3231.519886] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3231.535136] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3231.548741] LNet: 117427:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3232.609955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3233.113798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3233.134760] LNetError: 117480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3233.140947] LNetError: 117480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 3233.150280] LNet: Removed LNI 192.168.201.34@tcp [ 3233.168564] LNet: Removed LNI 192.168.201.34@tcp2 [ 3233.203725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3233.273923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3233.313729] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3234.314587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 3234.348677] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3235.652230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3235.752678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 3235.817711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 3235.912683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3236.366538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3237.407161] LNet: Removed LNI 192.168.201.34@tcp [ 3238.505898] LNet: Removed LNI 192.168.201.34@tcp2 [ 3238.612663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3238.732451] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3238.752545] LNet: Added LNI 192.168.201.34@tcp2 [8/256/0/180] [ 3240.113727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3250.857597] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 3253.606530] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:17:26 (1777497446) [ 3253.795888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3253.862570] LNet: Removed LNI 192.168.201.34@tcp [ 3254.889521] LNet: Removed LNI 192.168.201.34@tcp2 [ 3254.973573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3255.624786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3255.671953] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3256.970329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3257.022408] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3258.714555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3258.754420] LNet: Removed LNI 192.168.201.34@tcp [ 3258.766608] LNet: Removed LNI 10.1.2.3@tcp [ 3258.798583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3258.864425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 3258.889338] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3258.892811] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3259.274320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 3259.300942] LNet: Removed LNI 192.168.201.34@tcp [ 3259.310373] LNet: Removed LNI 10.1.2.3@tcp [ 3268.557862] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:17:40 (1777497460) [ 3268.727731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3268.794338] LNet: 119417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3268.810578] LNet: 119417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 3268.870086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3269.147921] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 3269.412104] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3269.432249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3269.451263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 3271.013395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3271.067711] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3272.982505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 3273.049779] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 3276.606954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.34@tcp [ 3276.720027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 3276.881744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.34@tcp -m PUT -e local_timeout -r 1 [ 3277.016497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 3277.218275] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 3277.226754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 3277.776034] LNetError: 119484:0:(lib-move.c:4969:lnet_parse()) 192.168.201.34@tcp, src 192.168.201.34@tcp: Dropping PUT (error -108 looking up sender) [ 3278.304379] LNetError: 119942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3278.322563] LNetError: 119942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 3278.342818] LNet: Removed LNI 192.168.201.34@tcp [ 3278.372769] LNet: Removed LNI 10.1.2.3@tcp [ 3279.411467] Key type .llcrypt unregistered [ 3279.422839] Key type ._llcrypt unregistered [ 3299.666292] Key type ._llcrypt registered [ 3299.667891] Key type .llcrypt registered [ 3301.892515] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:18:14 (1777497494) [ 3302.573506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3302.606110] alg: No test for adler32 (adler32-zlib) [ 3303.771437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3304.323619] LNet: 120793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3305.710413] Key type .llcrypt unregistered [ 3305.721612] Key type ._llcrypt unregistered [ 3314.454107] Key type ._llcrypt registered [ 3314.457060] Key type .llcrypt registered [ 3314.894945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3314.947162] alg: No test for adler32 (adler32-zlib) [ 3315.992888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3316.064478] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3325.621687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 3325.873418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 3326.662459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.134@tcp [ 3326.937566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3326.965110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 3327.001386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3327.035811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3327.111835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3327.159320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3327.259807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3327.315689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3327.331410] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.134@tcp [ 3327.369029] LNet: 121246:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3327.406675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3327.540147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3327.639643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3327.727697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3327.783707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3327.843812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3329.465067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3330.127892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3330.213587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 3330.266220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3330.326246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3330.430605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3330.512794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3330.643911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3330.746514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3330.792505] LNet: 121364:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3330.847266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3330.910952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3330.983287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3331.059664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3331.137738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3331.208888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3332.842871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.134@tcp [ 3333.573605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3333.732440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 3333.821693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3333.939040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3334.058956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3334.232026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3334.353071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3334.444800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3334.503597] LNet: 121481:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3334.539904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3334.634881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3334.689982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3334.742807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3334.825052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3334.939714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3336.695784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3337.082017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3337.131992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 3337.212131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3337.291763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3337.359751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3337.428606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3337.483725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3337.528649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3337.553708] LNet: 121599:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3337.580196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3337.628635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3337.697422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3337.758052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3337.813298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3337.903667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3338.621134] LNet: 121645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3338.641150] LNetError: 121645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3338.675431] LNet: Removed LNI 192.168.201.34@tcp [ 3339.478124] Key type .llcrypt unregistered [ 3339.482313] Key type ._llcrypt unregistered [ 3367.195593] Key type ._llcrypt registered [ 3367.200538] Key type .llcrypt registered [ 3368.905665] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:19:21 (1777497561) [ 3369.442403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3369.475128] alg: No test for adler32 (adler32-zlib) [ 3370.421735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3371.035288] LNet: 122534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3372.005975] Key type .llcrypt unregistered [ 3372.011287] Key type ._llcrypt unregistered [ 3381.935126] Key type ._llcrypt registered [ 3381.950569] Key type .llcrypt registered [ 3382.351827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3382.388691] alg: No test for adler32 (adler32-zlib) [ 3383.531863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 3383.568121] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3394.085599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.134@tcp [ 3407.380256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3407.456359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 3407.487709] LNet: Added LNI 192.168.201.34@tcp1 [8/256/0/180] [ 3408.672774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 3409.525680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.134@tcp [ 3409.797733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3409.824191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 3409.857910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3409.895225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3409.934661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3409.975227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3410.052710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3410.110669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3410.140635] LNet: 123197:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3410.171730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3410.209310] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.134@tcp [ 3410.282659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3410.342467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3410.381754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3410.413671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3410.485493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3411.274104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 3411.794624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3411.876022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 3411.938947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3412.005028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3412.080765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3412.123864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3412.168799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3412.205345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3412.226877] LNet: 123314:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3412.251315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3412.287377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3412.330079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3412.371238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3412.407417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3412.463663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3413.303890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.134@tcp [ 3413.586648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3413.619281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 3413.662519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3413.701155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3413.733363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3413.768633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3413.805616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3413.838594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3413.850394] LNet: 123431:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 3413.870505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3413.907408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3413.938189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3413.964697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3414.015791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3414.050965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3414.743458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 3415.071274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 3415.111463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 3415.151044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 3415.230494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 3415.363509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 3415.425441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 3415.492851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 3415.604614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 3415.689252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 3415.762736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 3415.801760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 3415.851752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 3415.961830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 3416.021484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 3422.900839] LNet: 123630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3422.914615] LNetError: 123630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3422.933170] LNet: Removed LNI 192.168.201.34@tcp [ 3422.955422] LNet: Removed LNI 192.168.201.34@tcp1 [ 3423.537675] Key type .llcrypt unregistered [ 3423.544239] Key type ._llcrypt unregistered [ 3449.046486] Key type ._llcrypt registered [ 3449.052362] Key type .llcrypt registered [ 3451.126668] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:20:43 (1777497643) [ 3461.917603] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:20:54 (1777497654) [ 3473.180484] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:21:05 (1777497665) [ 3485.148858] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:21:17 (1777497677) [ 3496.255709] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:21:28 (1777497688) [ 3508.796043] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:21:41 (1777497701) [ 3520.896279] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:21:53 (1777497713) [ 3532.425599] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:22:04 (1777497724) [ 3544.123643] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:22:16 (1777497736) [ 3554.930065] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:22:27 (1777497747) [ 3565.281670] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:22:38 (1777497758) [ 3575.284319] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:22:48 (1777497768) [ 3584.346530] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:22:57 (1777497777) [ 3594.116575] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:23:07 (1777497787) [ 3603.173873] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:23:15 (1777497795) [ 3612.481197] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:23:25 (1777497805) [ 3623.049773] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:23:35 (1777497815) [ 3633.682284] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:23:46 (1777497826) [ 3634.397694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3634.449383] alg: No test for adler32 (adler32-zlib) [ 3635.518868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3635.788459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3635.835044] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3637.774116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3637.825861] LNet: 134353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3637.845744] LNetError: 134353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3637.876901] LNet: Removed LNI 192.168.201.34@tcp [ 3637.911352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3638.065888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3638.093319] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3648.212253] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:24:00 (1777497840) [ 3658.121177] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:24:11 (1777497851) [ 3667.628541] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:24:20 (1777497860) [ 3677.147280] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:24:29 (1777497869) [ 3687.795549] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:24:40 (1777497880) [ 3698.302275] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:24:50 (1777497890) [ 3708.366136] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:25:01 (1777497901) [ 3718.949148] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:25:11 (1777497911) [ 3728.852262] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:25:21 (1777497921) [ 3738.800783] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:25:31 (1777497931) [ 3747.923823] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:25:40 (1777497940) [ 3757.550310] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:25:50 (1777497950) [ 3766.898511] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3439 sec ========== 17:25:59 (1777497959) [ 3767.438535] LNet: 141829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3767.446229] LNetError: 141829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3767.467537] LNet: Removed LNI 192.168.201.34@tcp [ 3768.402543] Key type .llcrypt unregistered [ 3768.412748] Key type ._llcrypt unregistered [ 3786.205968] Key type ._llcrypt registered [ 3786.222647] Key type .llcrypt registered [ 3787.007737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3787.022151] alg: No test for adler32 (adler32-zlib) [ 3788.261944] Lustre: Lustre: Build Version: 2.17.52_93_g6cdb58a [ 3788.599283] LNet: Added LNI 192.168.201.34@tcp [8/256/0/180] [ 3790.391194] Key type lgssc registered [ 3791.980715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3876.828253] Lustre: Mounted lustre-client [ 3883.406578] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3894.810162] LustreError: 144283:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff995a8a66e800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3894.905730] Lustre: Unmounted lustre-client [ 3926.958453] Key type lgssc unregistered [ 3927.312069] LNet: 144763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3927.320540] LNetError: 144763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3927.352877] LNet: Removed LNI 192.168.201.34@tcp [ 3928.288315] Key type .llcrypt unregistered [ 3928.293897] Key type ._llcrypt unregistered