[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 388290374 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002259] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.005925] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006008] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007004] pid_max: default: 32768 minimum: 301 [ 0.008076] LSM: Security Framework initializing [ 0.009027] Yama: becoming mindful. [ 0.010019] SELinux: Initializing. [ 0.011036] *** VALIDATE selinux *** [ 0.017616] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021150] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022098] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023065] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024058] *** VALIDATE tmpfs *** [ 0.025327] *** VALIDATE proc *** [ 0.026144] *** VALIDATE cgroup *** [ 0.027004] *** VALIDATE cgroup2 *** [ 0.028184] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.029103] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.030003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.031017] Spectre V2 : User space: Vulnerable [ 0.032013] Speculative Store Bypass: Vulnerable [ 0.034352] debug: unmapping init [mem 0xffffffffa5859000-0xffffffffa5860fff] [ 0.036154] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.037395] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.038009] ... version: 2 [ 0.038791] ... bit width: 48 [ 0.039005] ... generic registers: 4 [ 0.039733] ... value mask: 0000ffffffffffff [ 0.040005] ... max period: 00007fffffffffff [ 0.041005] ... fixed-purpose events: 3 [ 0.041717] ... event mask: 000000070000000f [ 0.042130] rcu: Hierarchical SRCU implementation. [ 0.044075] smp: Bringing up secondary CPUs ... [ 0.045250] x86: Booting SMP configuration: [ 0.046015] .... node #0, CPUs: #1 #2 #3 [ 0.048623] smp: Brought up 1 node, 4 CPUs [ 0.049862] smpboot: Max logical packages: 1 [ 0.050006] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.255907] node 0 deferred pages initialised in 204ms [ 0.258128] devtmpfs: initialized [ 0.259170] x86/mm: Memory block size: 128MB [ 0.261310] gcov: version magic: 0x41383552 [ 0.262554] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.263050] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.264151] pinctrl core: initialized pinctrl subsystem [ 0.265092] [ 0.265413] ************************************************************* [ 0.266006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267005] ** ** [ 0.268006] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.269007] ** ** [ 0.270006] ** This means that this kernel is built to expose internal ** [ 0.271008] ** IOMMU data structures, which may compromise security on ** [ 0.272006] ** your system. ** [ 0.273005] ** ** [ 0.274007] ** If you see this message and you are not debugging the ** [ 0.275005] ** kernel, report this immediately to your vendor! ** [ 0.276005] ** ** [ 0.277007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278005] ************************************************************* [ 0.279494] NET: Registered protocol family 16 [ 0.280308] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.281025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.282025] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.283276] cpuidle: using governor menu [ 0.284442] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.286383] PCI: Using configuration type 1 for base access [ 0.287101] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.293120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.294010] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.295266] cryptd: max_cpu_qlen set to 1000 [ 0.296173] ACPI: Added _OSI(Module Device) [ 0.297000] ACPI: Added _OSI(Processor Device) [ 0.298010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.299006] ACPI: Added _OSI(Processor Aggregator Device) [ 0.302570] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.307190] ACPI: Interpreter enabled [ 0.308023] ACPI: PM: (supports S0 S3 S4 S5) [ 0.308897] ACPI: Using IOAPIC for interrupt routing [ 0.309054] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.312340] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.320331] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.321017] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.322010] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.325039] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.327873] acpiphp: Slot [2] registered [ 0.329061] acpiphp: Slot [5] registered [ 0.329970] acpiphp: Slot [6] registered [ 0.330057] acpiphp: Slot [3] registered [ 0.330973] acpiphp: Slot [4] registered [ 0.332045] acpiphp: Slot [7] registered [ 0.332898] acpiphp: Slot [8] registered [ 0.334055] acpiphp: Slot [9] registered [ 0.334917] acpiphp: Slot [10] registered [ 0.335042] acpiphp: Slot [11] registered [ 0.335870] acpiphp: Slot [12] registered [ 0.337057] acpiphp: Slot [13] registered [ 0.337822] acpiphp: Slot [14] registered [ 0.339048] acpiphp: Slot [15] registered [ 0.339884] acpiphp: Slot [16] registered [ 0.340046] acpiphp: Slot [17] registered [ 0.341094] acpiphp: Slot [18] registered [ 0.342060] acpiphp: Slot [19] registered [ 0.342967] acpiphp: Slot [20] registered [ 0.344069] acpiphp: Slot [21] registered [ 0.345015] acpiphp: Slot [22] registered [ 0.346009] acpiphp: Slot [23] registered [ 0.346987] acpiphp: Slot [24] registered [ 0.348047] acpiphp: Slot [25] registered [ 0.348985] acpiphp: Slot [26] registered [ 0.350047] acpiphp: Slot [27] registered [ 0.351060] acpiphp: Slot [28] registered [ 0.352021] acpiphp: Slot [29] registered [ 0.352928] acpiphp: Slot [30] registered [ 0.354046] acpiphp: Slot [31] registered [ 0.355056] PCI host bridge to bus 0000:00 [ 0.355965] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.357009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.359013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.361014] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.364017] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.366040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.367169] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.371177] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.375350] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.382012] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.384521] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.386008] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.388012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.391012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.394887] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.398824] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.402049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.404624] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.408973] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.416615] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.420007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.423410] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.427973] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.432007] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.440832] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.447971] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.450007] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.453007] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.463010] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.469000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.471214] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.473203] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.474183] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.475324] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.480220] iommu: Default domain type: Passthrough [ 0.481306] SCSI subsystem initialized [ 0.482057] ACPI: bus type USB registered [ 0.483059] usbcore: registered new interface driver usbfs [ 0.484032] usbcore: registered new interface driver hub [ 0.486058] usbcore: registered new device driver usb [ 0.487090] pps_core: LinuxPPS API ver. 1 registered [ 0.488006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.490019] PTP clock support registered [ 0.491116] EDAC MC: Ver: 3.0.0 [ 0.493169] PCI: Using ACPI for IRQ routing [ 0.494793] NetLabel: Initializing [ 0.496015] NetLabel: domain hash size = 128 [ 0.498009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.499092] NetLabel: unlabeled traffic allowed by default [ 0.501139] vgaarb: loaded [ 0.503000] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.504008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.508230] clocksource: Switched to clocksource kvm-clock [ 0.590656] VFS: Disk quotas dquot_6.6.0 [ 0.591711] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.593370] *** VALIDATE ramfs *** [ 0.594182] *** VALIDATE hugetlbfs *** [ 0.595199] pnp: PnP ACPI init [ 0.597450] pnp: PnP ACPI: found 6 devices [ 0.610326] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.612473] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.613988] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.615472] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.617083] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.618841] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.620691] NET: Registered protocol family 2 [ 0.622479] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.626049] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.628449] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.632480] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.634800] TCP: Hash tables configured (established 65536 bind 65536) [ 0.636655] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.638678] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.640530] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.642398] NET: Registered protocol family 1 [ 0.644177] RPC: Registered named UNIX socket transport module. [ 0.646319] RPC: Registered udp transport module. [ 0.647965] RPC: Registered tcp transport module. [ 0.649361] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.651135] NET: Registered protocol family 44 [ 0.652797] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.654821] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.656876] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.659199] PCI: CLS 0 bytes, default 64 [ 0.660819] Unpacking initramfs... [ 1.958204] debug: unmapping init [mem 0xffff8c88bcc64000-0xffff8c88bffcffff] [ 1.961486] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.963851] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.966424] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.429464] Initialise system trusted keyrings [ 2.430911] Key type blacklist registered [ 2.432310] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.440929] zbud: loaded [ 2.443790] *** VALIDATE nfs *** [ 2.444535] *** VALIDATE nfs4 *** [ 2.445551] pstore: using deflate compression [ 2.447266] Platform Keyring initialized [ 2.515482] NET: Registered protocol family 38 [ 2.516749] Key type asymmetric registered [ 2.517620] Asymmetric key parser 'x509' registered [ 2.518664] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.520532] io scheduler mq-deadline registered [ 2.521544] io scheduler kyber registered [ 2.522542] io scheduler bfq registered [ 2.523727] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.525538] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.527564] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.529272] ACPI: Power Button [PWRF] [ 2.532889] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.537636] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.543875] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.569475] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.596021] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.600904] Non-volatile memory driver v1.3 [ 2.603428] Linux agpgart interface v0.103 [ 2.632061] virtio_blk virtio1: [vda] 134144 512-byte logical blocks (68.7 MB/65.5 MiB) [ 2.636223] vda: detected capacity change from 0 to 68681728 [ 2.652942] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.655986] vdb: detected capacity change from 0 to 1073741824 [ 2.661562] libphy: Fixed MDIO Bus: probed [ 2.665877] usbcore: registered new interface driver usbserial_generic [ 2.668477] usbserial: USB Serial support registered for generic [ 2.671405] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.676310] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.678351] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.681060] mousedev: PS/2 mouse device common for all mice [ 2.684370] rtc_cmos 00:05: RTC can wake from S4 [ 2.686887] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.688195] rtc_cmos 00:05: registered as rtc0 [ 2.693463] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.694202] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.696676] intel_pstate: CPU model not supported [ 2.701884] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.706976] hid: raw HID events driver (C) Jiri Kosina [ 2.708303] usbcore: registered new interface driver usbhid [ 2.710733] usbhid: USB HID core driver [ 2.712502] drop_monitor: Initializing network drop monitor service [ 2.715292] Initializing XFRM netlink socket [ 2.717618] NET: Registered protocol family 10 [ 2.721177] Segment Routing with IPv6 [ 2.722975] NET: Registered protocol family 17 [ 2.725260] mpls_gso: MPLS GSO support [ 2.730629] RAS: Correctable Errors collector initialized. [ 2.733213] AVX version of gcm_enc/dec engaged. [ 2.735475] AES CTR mode by8 optimization enabled [ 2.794228] sched_clock: Marking stable (2794206988, 0)->(3446602353, -652395365) [ 2.796724] registered taskstats version 1 [ 2.798273] Loading compiled-in X.509 certificates [ 2.800393] zswap: loaded using pool lzo/zbud [ 2.818873] Key type big_key registered [ 2.828437] Key type encrypted registered [ 2.829509] ima: No TPM chip found, activating TPM-bypass! [ 2.830974] ima: Allocated hash algorithm: sha1 [ 2.832217] ima: No architecture policies found [ 2.833436] evm: Initialising EVM extended attributes: [ 2.834774] evm: security.selinux [ 2.835605] evm: security.ima [ 2.836473] evm: security.capability [ 2.837386] evm: HMAC attrs: 0x1 [ 2.839354] rtc_cmos 00:05: setting system clock to 2026-05-09 19:26:24 UTC (1778354784) [ 2.845688] debug: unmapping init [mem 0xffffffffa6803000-0xffffffffa69fffff] [ 2.849056] debug: unmapping init [mem 0xffffffffa5582000-0xffffffffa5858fff] [ 2.858057] Write protecting the kernel read-only data: 28672k [ 2.861126] debug: unmapping init [mem 0xffffffffa3c03000-0xffffffffa3dfffff] [ 2.862900] debug: unmapping init [mem 0xffffffffa4514000-0xffffffffa45fffff] [ 2.887953] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.894560] systemd[1]: Detected virtualization kvm. [ 2.896196] systemd[1]: Detected architecture x86-64. [ 2.897739] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.923249] systemd[1]: No hostname configured. [ 2.925154] systemd[1]: Set hostname to . [ 2.927670] random: systemd: uninitialized urandom read (16 bytes read) [ 2.930388] systemd[1]: Initializing machine ID from random generator. [ 2.983088] random: ln: uninitialized urandom read (6 bytes read) [ 3.078495] random: systemd: uninitialized urandom read (16 bytes read) [ 3.081270] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.085908] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.096691] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.611243] device-mapper: uevent: version 1.0.3 [ 3.612572] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.257701] virtio_net virtio0 ens2: renamed from eth0 [ 4.294641] scsi host0: ata_piix [ 4.308983] scsi host1: ata_piix [ 4.310265] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.312060] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.610277] random: fast init done [ 8.677725] dracut-initqueue[593]: RTNETLINK answers: File exists [ 9.342353] random: crng init done [ 9.344212] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.752607] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.087839] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.401836] SELinux: Disabled at runtime. [ 11.473983] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.480186] systemd[1]: Detected virtualization kvm. [ 11.481552] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.965826] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.968908] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.973217] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.979848] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.983303] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.989341] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.995890] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ 12.058369] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.605757] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.952214] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.079155] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.093953] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.107958] EDAC sbridge: Ver: 1.1.2 [ 14.240573] Key type dns_resolver registered [ 14.560369] NFS: Registering the id_resolver key type [ 14.562402] Key type id_resolver registered [ 14.564361] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-client login: [ 40.644673] libcfs: loading out-of-tree module taints kernel. [ 40.765237] Key type ._llcrypt registered [ 40.766419] Key type .llcrypt registered [ 41.005819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.010684] alg: No test for adler32 (adler32-zlib) [ 42.015682] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 42.289110] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 43.911227] Key type lgssc registered [ 44.499339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 87.446199] Lustre: Mounted lustre-client [ 89.941262] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 102.554957] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing check_logdir /tmp/testlogs/ [ 104.052961] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing yml_node [ 105.470306] Lustre: DEBUG MARKER: Client: 2.17.52.4 [ 106.714696] Lustre: DEBUG MARKER: MDS: 2.17.52.4 [ 107.976052] Lustre: DEBUG MARKER: OSS: 2.17.52.4 [ 108.741656] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat May 9 15:28:09 EDT 2026 [ 113.119386] Lustre: lustre-OST0000-osc-ffff8c891909f800: disconnect after 24s idle [ 114.953428] LustreError: 7905:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8c891909f800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 114.968446] Lustre: Unmounted lustre-client [ 124.267150] Key type lgssc unregistered [ 124.387398] LNet: 8349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 124.389985] LNetError: 8349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 124.397808] LNet: Removed LNI 192.168.203.16@tcp [ 124.682154] Key type .llcrypt unregistered [ 124.683445] Key type ._llcrypt unregistered [ 133.151766] Key type ._llcrypt registered [ 133.153358] Key type .llcrypt registered [ 133.218673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 133.224591] alg: No test for adler32 (adler32-zlib) [ 134.037329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 134.064264] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 134.072796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 135.440466] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 135.472809] LNet: 9220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 135.475844] LNetError: 9220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 135.483575] LNet: Removed LNI 192.168.203.16@tcp [ 135.742300] Key type .llcrypt unregistered [ 135.743516] Key type ._llcrypt unregistered [ 137.870341] Key type ._llcrypt registered [ 137.871362] Key type .llcrypt registered [ 138.339131] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:28:39 (1778354919) [ 138.508560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 138.515480] alg: No test for adler32 (adler32-zlib) [ 139.320026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 139.351189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 139.359803] LNet: 9636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 139.366791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 139.384643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 139.391620] LNet: 9643:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 141.691479] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:28:42 (1778354922) [ 141.721707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 141.731022] LNet: 10214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 141.741691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 141.774061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 144.063870] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:28:45 (1778354925) [ 144.106759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 144.117366] LNet: 10800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 144.128385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 144.164902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 146.394160] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:28:47 (1778354927) [ 146.442028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 146.452499] LNet: 11386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 146.464122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 146.495802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 148.755256] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:28:49 (1778354929) [ 148.805868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 148.825391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 148.862136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 148.994458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 151.206731] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:28:52 (1778354932) [ 151.244660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 151.255157] LNet: 12557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 151.259014] LNet: 12557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 151.274382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 151.307767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 151.468838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 151.616181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 153.883953] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:28:55 (1778354935) [ 153.925098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 153.941921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 153.978242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 156.253756] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:28:57 (1778354937) [ 156.302802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 156.322683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 156.349897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 156.361785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 156.501794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 156.518588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 156.698754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 156.717339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 156.888169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 156.905987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 157.045023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 157.062143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 157.204047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 157.218609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 157.364365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 157.379777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 157.519315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 157.532533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 157.678972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 157.695042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 157.860746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 157.875956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 158.021841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 158.040432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 158.233194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 158.251237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 158.400789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 158.415398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 160.558267] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:29:01 (1778354941) [ 160.589818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 160.601086] LNet: 14416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 160.603404] LNet: 14416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 160.613037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 160.642425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 160.654204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 162.692198] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:29:03 (1778354943) [ 162.727795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 162.746678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 162.779034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 162.797584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 165.038746] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:29:06 (1778354946) [ 165.071537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 165.090351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 165.127211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 165.142428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 167.420746] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:29:08 (1778354948) [ 167.453994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 167.472912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 167.507519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 167.520339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 169.824094] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:29:10 (1778354950) [ 169.860840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 169.877692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 169.907140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 169.924931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 172.215786] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:29:13 (1778354953) [ 172.255156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 172.272642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 172.305796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 172.319197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 174.526655] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:29:15 (1778354955) [ 174.575678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 174.591043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 174.614826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 174.726148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 176.681677] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:29:17 (1778354957) [ 176.710091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 176.719156] LNet: 18521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 176.721702] LNet: 18521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 176.731389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 176.806246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 176.892336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 178.977034] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:29:20 (1778354960) [ 179.007221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 179.021981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 179.064273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 179.140223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 181.193933] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:29:22 (1778354962) [ 181.246764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 181.263846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 181.406348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 181.484493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 183.461250] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:29:24 (1778354964) [ 183.500234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 183.520125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 183.552266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 183.572304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 185.479788] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:29:26 (1778354966) [ 185.509424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 185.524979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 185.553167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 185.567386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 187.632013] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:29:28 (1778354968) [ 187.673760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 187.690209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 187.711989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 189.740833] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:29:30 (1778354970) [ 189.787245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.802936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.827525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 191.741295] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:29:33 (1778354973) [ 191.765284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.778074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.806505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 191.923075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 193.975761] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:29:35 (1778354975) [ 194.030476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 194.042947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 194.067024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 194.079619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 196.116117] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:29:37 (1778354977) [ 196.151264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.172847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.208925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 196.221447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 198.398876] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:29:39 (1778354979) [ 198.447202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 198.462688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 198.484078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 198.495795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 200.610874] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:29:41 (1778354981) [ 200.648612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 200.670427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 200.701092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 200.716539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 202.841527] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:29:44 (1778354984) [ 202.893589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.910490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.934098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 202.958128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 202.966859] LNetError: 25720:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 202.980501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 202.991062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 204.859237] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:29:46 (1778354986) [ 204.901588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 204.917501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.947540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JDIQ8xUijh [ 204.975162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JDIQ8xUijh [ 204.993778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.JDIQ8xUijh [ 206.863464] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:29:48 (1778354988) [ 206.896463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.908345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.928066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 206.938574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 206.954288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 208.790235] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:29:50 (1778354990) [ 208.822593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 208.831180] LNet: 27472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 208.833555] LNet: 27472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 208.845251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 208.870652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 209.823552] LNet: Message forwarding will be enabled in 2s [ 212.895134] LNet: Message forwarding enabled [ 212.948463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 212.954499] LNet: Message forwarding will stop in 2s to 6s [ 217.079488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 217.571405] LNet: Message forwarding disabled [ 217.957823] LNet: Message forwarding will be enabled in 2s [ 219.827557] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:30:01 (1778355001) [ 219.865888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.350662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.374327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 222.350657] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:30:03 (1778355003) [ 222.381735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 222.396235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 222.419701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 222.430668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 222.441877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 222.453737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 222.462906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 222.472223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 222.482645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 222.496133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 222.511789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 222.522145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 222.533942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 222.545172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 222.556703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 222.566732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 222.577373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 222.589055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 222.600476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 222.614347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 222.629510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 222.638943] LNetError: 28785:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 224.418797] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:30:05 (1778355005) [ 224.457540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 224.476660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 224.505487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 226.541631] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:30:07 (1778355007) [ 227.147525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg316-server [ 227.750627] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:30:08 (1778355008) [ 227.788994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.808492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.841627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 227.851767] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 228.457224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[17-21]@tcp [ 228.466855] LNet: 30499:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 228.476937] LNetError: 30288:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.19@tcp is being used as a gateway but routing feature is not turned on [ 228.482180] LNetError: 30288:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.19@tcp has gone from up to down [ 229.537346] LNetError: 30288:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.19@tcp is being used as a gateway but routing feature is not turned on [ 229.543462] LNetError: 30288:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 230.561747] LNetError: 30288:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.18@tcp is being used as a gateway but routing feature is not turned on [ 230.567345] LNetError: 30288:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 230.647425] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:30:11 (1778355011) [ 231.259540] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg316-server [ 231.519287] LNetError: 30293:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.17@tcp has gone from up to down [ 231.523394] LNetError: 30293:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 231.815260] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:30:13 (1778355013) [ 231.861388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 231.873550] LNetError: 31433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 231.881778] LNet: Removed LNI 192.168.203.16@tcp [ 231.888911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 231.909703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 231.918715] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 232.143770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[17-22/2]@tcp [ 232.151296] LNet: 31550:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 232.154575] LNet: 31550:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 232.162948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[17-22/2]@tcp [ 234.245401] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:30:15 (1778355015) [ 234.284732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.359717] LNetError: 32140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 235.366515] LNet: Removed LNI 192.168.203.16@tcp [ 235.373021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.401861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 235.421608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.437497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.466239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.491054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.506818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.531252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 235.559274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.571179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.592441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.618972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.632713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.652568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 235.678862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.692220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.718525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.753622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.778363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.802667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 235.837666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.856243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.881901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 235.911870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.926388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.947245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 235.976519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.987944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.007955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 236.035862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.048305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.066819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 236.088535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.103601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.125632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 238.286521] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:30:19 (1778355019) [ 238.965530] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg316-server [ 239.529830] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:30:20 (1778355020) [ 240.147317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg316-server [ 240.704540] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:30:21 (1778355021) [ 241.009393] Key type .llcrypt unregistered [ 241.010448] Key type ._llcrypt unregistered [ 243.090734] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 243.136934] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 243.138600] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 243.140981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 245.370335] Key type ._llcrypt registered [ 245.371293] Key type .llcrypt registered [ 245.447171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 245.452654] alg: No test for adler32 (adler32-zlib) [ 246.257786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.278206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 246.288878] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 246.536416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 246.546156] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 247.273709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 247.283071] LNet: Removed LNI 10.1.2.3@tcp [ 247.324673] LNet: 34260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 247.327112] LNetError: 34260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 247.335508] LNet: Removed LNI 192.168.203.16@tcp [ 247.555519] Key type .llcrypt unregistered [ 247.556713] Key type ._llcrypt unregistered [ 251.257430] Key type ._llcrypt registered [ 251.259255] Key type .llcrypt registered [ 251.708566] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:30:32 (1778355032) [ 251.880426] Key type .llcrypt unregistered [ 251.881584] Key type ._llcrypt unregistered [ 253.741548] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 253.787399] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 253.789271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 253.790910] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 256.000303] Key type ._llcrypt registered [ 256.001919] Key type .llcrypt registered [ 256.065789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 256.071279] alg: No test for adler32 (adler32-zlib) [ 256.870869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 256.892072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 256.902405] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 257.153360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 257.161780] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 258.003804] LNet: 35749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.008052] LNetError: 35749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 258.012388] LNet: Removed LNI 192.168.203.16@tcp [ 258.019352] LNet: Removed LNI 10.1.2.3@tcp [ 258.246273] Key type .llcrypt unregistered [ 258.247269] Key type ._llcrypt unregistered [ 261.786985] Key type ._llcrypt registered [ 261.788487] Key type .llcrypt registered [ 262.206414] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:30:43 (1778355043) [ 262.374224] Key type .llcrypt unregistered [ 262.375565] Key type ._llcrypt unregistered [ 264.175198] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 264.220554] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 264.223292] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 264.226336] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 266.433032] Key type ._llcrypt registered [ 266.434127] Key type .llcrypt registered [ 266.505795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 266.513635] alg: No test for adler32 (adler32-zlib) [ 267.317523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.342319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 267.354610] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 268.073229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 268.084634] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 268.840977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 268.850721] LNet: Removed LNI 10.1.2.3@tcp [ 268.856496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 268.873435] LNet: Removed LNI 10.1.2.31@tcp [ 268.879874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 268.888031] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 269.152398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 269.159876] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 269.512184] LNet: 37665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 269.515037] LNetError: 37665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 269.519154] LNet: Removed LNI 10.1.2.3@tcp [ 269.525361] LNet: Removed LNI 10.1.2.31@tcp [ 269.765407] Key type .llcrypt unregistered [ 269.766911] Key type ._llcrypt unregistered [ 273.774701] Key type ._llcrypt registered [ 273.775926] Key type .llcrypt registered [ 274.246229] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:30:55 (1778355055) [ 274.368593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 274.373772] alg: No test for adler32 (adler32-zlib) [ 275.182883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.203332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 275.211831] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 275.514225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.522980] LNet: 38617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 275.526894] LNetError: 38617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 275.539301] LNet: Removed LNI 192.168.203.16@tcp [ 275.548877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.567712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.16@tcp [ 275.576174] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 275.821125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.16@tcp [ 275.834301] LNet: Removed LNI 192.168.203.16@tcp [ 276.071434] Key type .llcrypt unregistered [ 276.072637] Key type ._llcrypt unregistered [ 279.760203] Key type ._llcrypt registered [ 279.761423] Key type .llcrypt registered [ 280.213983] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:31:01 (1778355061) [ 280.823786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg316-server [ 281.351448] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:31:02 (1778355062) [ 281.512389] Key type .llcrypt unregistered [ 281.514212] Key type ._llcrypt unregistered [ 283.570521] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 283.628262] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 283.630280] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 283.632477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 285.817594] Key type ._llcrypt registered [ 285.818517] Key type .llcrypt registered [ 285.874573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 285.879358] alg: No test for adler32 (adler32-zlib) [ 286.677341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.699535] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 286.942335] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 287.661769] LNet: 40585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 287.664413] LNetError: 40585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 287.666789] LNet: Removed LNI 192.168.203.16@tcp1 [ 287.674290] LNet: Removed LNI 10.1.2.3@tcp2 [ 287.903358] Key type .llcrypt unregistered [ 287.904617] Key type ._llcrypt unregistered [ 291.710820] Key type ._llcrypt registered [ 291.711885] Key type .llcrypt registered [ 292.158412] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:31:13 (1778355073) [ 292.293746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 292.299229] alg: No test for adler32 (adler32-zlib) [ 293.108048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.134337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 293.147099] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 293.423390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.431561] LNet: 41551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.434598] LNetError: 41551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 293.443437] LNet: Removed LNI 192.168.203.16@tcp [ 293.450248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.472167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 293.482090] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 295.542256] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:31:16 (1778355076) [ 295.586172] LNet: 42139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.588310] LNetError: 42139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 295.599894] LNet: Removed LNI 192.168.203.16@tcp [ 295.830458] Key type .llcrypt unregistered [ 295.831678] Key type ._llcrypt unregistered [ 297.827105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 297.904321] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 297.905992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 297.909840] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 300.096176] Key type ._llcrypt registered [ 300.097303] Key type .llcrypt registered [ 300.169298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 300.176960] alg: No test for adler32 (adler32-zlib) [ 300.986402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.014308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 301.025746] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 301.312899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 301.319910] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 302.142060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.149832] LNet: 42882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.152331] LNetError: 42882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.154678] LNet: Removed LNI 192.168.203.16@tcp [ 302.162509] LNet: Removed LNI 10.1.2.3@tcp [ 302.170970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.199492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 302.209141] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 302.210897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 302.375497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.385652] LNet: Removed LNI 192.168.203.16@tcp [ 302.394436] LNet: Removed LNI 10.1.2.3@tcp [ 302.403558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.435384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 302.447067] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 302.448952] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 302.583676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 302.591502] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 302.983282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.990639] LNet: 43072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.992670] LNet: 43072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 302.994996] LNetError: 43072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.997053] LNetError: 43072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 302.999162] LNet: Removed LNI 192.168.203.16@tcp [ 303.000293] LNet: Removed LNI 10.1.2.3@tcp [ 303.007363] LNet: Removed LNI 192.168.203.16@tcp2 [ 303.016141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.043876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 303.058210] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 303.061298] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 303.065451] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 305.402905] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:31:26 (1778355086) [ 305.446969] LNet: 43662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.449626] LNetError: 43662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 305.452086] LNetError: 43662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 305.454254] LNet: Removed LNI 192.168.203.16@tcp [ 305.455513] LNet: Removed LNI 10.1.2.3@tcp [ 305.461836] LNet: Removed LNI 192.168.203.16@tcp2 [ 305.694569] Key type .llcrypt unregistered [ 305.695914] Key type ._llcrypt unregistered [ 307.815285] Key type ._llcrypt registered [ 307.817200] Key type .llcrypt registered [ 307.883647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 307.889731] alg: No test for adler32 (adler32-zlib) [ 308.692387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.709790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 308.717548] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 309.006554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.013116] LNet: 44062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.015692] LNetError: 44062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.022343] LNet: Removed LNI 192.168.203.16@tcp [ 309.030296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.060933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 309.069408] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 309.201365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.213719] LNet: Removed LNI 192.168.203.16@tcp [ 309.222142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.252210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 309.262849] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 309.401335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.417887] LNet: Removed LNI 192.168.203.16@tcp [ 309.424539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.447514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 309.457170] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 309.589541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 309.645243] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 309.647492] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 309.649777] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 311.780069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.789817] LNet: 44201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.792164] LNet: 44201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 311.794614] LNetError: 44201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 311.796731] LNetError: 44201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 311.806323] LNet: Removed LNI 192.168.203.16@tcp [ 311.813420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.835190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 311.844149] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 314.079188] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:31:35 (1778355095) [ 314.124728] LNet: 44792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.127446] LNetError: 44792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 314.134127] LNet: Removed LNI 192.168.203.16@tcp [ 314.389788] Key type .llcrypt unregistered [ 314.390979] Key type ._llcrypt unregistered [ 316.509682] Key type ._llcrypt registered [ 316.510841] Key type .llcrypt registered [ 316.592490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 316.598645] alg: No test for adler32 (adler32-zlib) [ 317.409930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.432892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 317.442094] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 317.789961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.797517] LNet: 45194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.800558] LNetError: 45194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.809327] LNet: Removed LNI 192.168.203.16@tcp [ 317.817553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.837925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 317.846477] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 319.987519] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:31:41 (1778355101) [ 320.029621] LNet: 45779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 320.033274] LNetError: 45779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 320.043896] LNet: Removed LNI 192.168.203.16@tcp [ 320.296583] Key type .llcrypt unregistered [ 320.297718] Key type ._llcrypt unregistered [ 322.258342] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 322.313778] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 322.315234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 322.316966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 324.495249] Key type ._llcrypt registered [ 324.496161] Key type .llcrypt registered [ 324.553629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 324.559138] alg: No test for adler32 (adler32-zlib) [ 325.363690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.381248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 325.389780] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 325.678700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.687242] LNet: 46228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.691181] LNetError: 46228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 325.702326] LNet: Removed LNI 192.168.203.16@tcp [ 325.709680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.732977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 325.744863] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 327.880321] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:31:49 (1778355109) [ 327.922047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.929820] LNet: 46824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.932366] LNetError: 46824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 327.939689] LNet: Removed LNI 192.168.203.16@tcp [ 327.948732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.988293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 330.050315] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:31:51 (1778355111) [ 330.082199] LNet: 47401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.304512] Key type .llcrypt unregistered [ 330.305607] Key type ._llcrypt unregistered [ 332.236775] Key type ._llcrypt registered [ 332.237706] Key type .llcrypt registered [ 332.299641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 332.306094] alg: No test for adler32 (adler32-zlib) [ 333.110736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.135583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 333.147985] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 333.460033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.469538] LNet: 47804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.473251] LNetError: 47804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 333.482538] LNet: Removed LNI 192.168.203.16@tcp [ 333.491527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.521408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 333.532532] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 335.686893] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:31:56 (1778355116) [ 335.719950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.728619] LNet: 48392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.731159] LNetError: 48392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 335.738926] LNet: Removed LNI 192.168.203.16@tcp [ 335.749507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.774761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 337.707419] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:31:58 (1778355118) [ 338.184336] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 338.743036] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:31:59 (1778355119) [ 339.244567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 339.789900] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:32:00 (1778355120) [ 340.285176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 340.820917] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:32:02 (1778355122) [ 341.288585] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 341.782326] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:32:03 (1778355123) [ 342.311302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 342.843457] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:32:04 (1778355124) [ 342.881056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.893137] LNet: 50446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.906208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.932732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 342.943820] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 343.218855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.16 [ 343.226219] LNet: 50582:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 343.229795] LNet: Removed LNI 192.168.203.16@tcp [ 345.272230] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:32:06 (1778355126) [ 345.307228] LNet: 51145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.309988] LNetError: 51145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 345.316624] LNet: Removed LNI 192.168.203.16@tcp [ 345.560745] Key type .llcrypt unregistered [ 345.562174] Key type ._llcrypt unregistered [ 347.656661] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 347.766201] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 347.767880] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 347.770049] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 350.032357] Key type ._llcrypt registered [ 350.033505] Key type .llcrypt registered [ 350.084597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 350.089391] alg: No test for adler32 (adler32-zlib) [ 351.041989] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 353.097852] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:32:14 (1778355134) [ 353.150653] LNet: 52055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.154539] LNetError: 52055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 353.164358] LNet: Removed LNI 10.1.2.3@tcp [ 353.376409] Key type .llcrypt unregistered [ 353.378315] Key type ._llcrypt unregistered [ 355.292409] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 355.391180] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 355.393676] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 355.395649] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 357.659888] Key type ._llcrypt registered [ 357.660949] Key type .llcrypt registered [ 357.711965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 357.716980] alg: No test for adler32 (adler32-zlib) [ 358.530736] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 360.571446] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:32:21 (1778355141) [ 360.604850] LNet: 52939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.607488] LNetError: 52939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 360.615850] LNet: Removed LNI 10.1.2.3@tcp [ 360.852294] Key type .llcrypt unregistered [ 360.853471] Key type ._llcrypt unregistered [ 362.908153] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 363.032147] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 363.033612] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 363.035239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 365.315257] Key type ._llcrypt registered [ 365.316297] Key type .llcrypt registered [ 365.382436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 365.388215] alg: No test for adler32 (adler32-zlib) [ 366.217698] LNet: 53246:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 366.222832] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 368.322122] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:32:29 (1778355149) [ 368.360842] LNet: 53833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.363703] LNetError: 53833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 368.372512] LNet: Removed LNI 10.1.2.3@tcp [ 368.579365] Key type .llcrypt unregistered [ 368.580471] Key type ._llcrypt unregistered [ 370.516337] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 370.604296] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 370.605950] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 370.607612] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 372.865092] Key type ._llcrypt registered [ 372.866300] Key type .llcrypt registered [ 372.927791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 372.934478] alg: No test for adler32 (adler32-zlib) [ 373.737671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 373.769468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 376.512237] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:32:37 (1778355157) [ 376.687385] LNet: 55065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.692388] LNetError: 55065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 376.704651] LNet: Removed LNI 10.1.2.3@tcp [ 376.930648] Key type .llcrypt unregistered [ 376.932435] Key type ._llcrypt unregistered [ 378.957390] Key type ._llcrypt registered [ 378.958497] Key type .llcrypt registered [ 379.032854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 379.038783] alg: No test for adler32 (adler32-zlib) [ 379.850255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 379.862249] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 382.705688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 382.918933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 382.931288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 382.943765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 382.950544] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 383.032373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 383.047711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 383.062979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 383.078399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 383.094489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 383.110589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 383.125063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 383.141222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 383.150716] LNet: 55501:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 383.157745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 383.170031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 383.180692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 383.194500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 383.208465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 383.220836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 383.233125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 383.412362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 383.425709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 383.439958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 383.519863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 383.530635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 383.541864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 383.553255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 383.565816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 383.579714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 383.591102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 383.602534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 383.614482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 383.628061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 383.640724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 383.653858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 383.667270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 383.680336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 383.691937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 383.863631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 383.876267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 383.889358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 383.897195] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 383.899477] LNet: Skipped 1 previous similar message [ 383.967246] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 383.979798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 383.992805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 384.006921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 384.020095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 384.031465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 384.042695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 384.054374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 384.065686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 384.072421] LNet: 55719:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 384.074431] LNet: 55719:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 384.081716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 384.094857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 384.106895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 384.118347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 384.130728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 384.143389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 384.155185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 384.329077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 384.342705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 384.356862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 384.434831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 384.447287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 384.458723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 384.468742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 384.481565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 384.494668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 384.507269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 384.521073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 384.535501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 384.549397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 384.562329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 384.574682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 384.587448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 384.601885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 384.615325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 384.793503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 384.811979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 384.829156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 384.912595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 384.928016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 384.940369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 384.953259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 384.965867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 384.977797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 384.988875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 385.000362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 385.011730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 385.023526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 385.037957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 385.049576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 385.064630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 385.076872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 385.090087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 385.261992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 385.275477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 385.286732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 385.293083] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 385.296015] LNet: Skipped 3 previous similar messages [ 385.367289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 385.378558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 385.390641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 385.403217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 385.414923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 385.427859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 385.440784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 385.451988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 385.458414] LNet: 56046:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 385.460700] LNet: 56046:0:(api-ni.c:296:recovery_interval_set()) Skipped 2 previous similar messages [ 385.467712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 385.479683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 385.490434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 385.501829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 385.513245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 385.526117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 385.538855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 385.621321] LNet: 56087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.623725] LNetError: 56087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 385.636348] LNet: Removed LNI 192.168.203.16@tcp [ 385.769423] Key type .llcrypt unregistered [ 385.770551] Key type ._llcrypt unregistered [ 391.000897] Key type ._llcrypt registered [ 391.002640] Key type .llcrypt registered [ 391.507713] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:32:52 (1778355172) [ 391.815505] Key type .llcrypt unregistered [ 391.817739] Key type ._llcrypt unregistered [ 393.834644] Key type ._llcrypt registered [ 393.836503] Key type .llcrypt registered [ 393.897078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 393.902660] alg: No test for adler32 (adler32-zlib) [ 394.710510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 394.722807] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 397.528563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 400.494721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.511572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 400.522040] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 401.037790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 401.058326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 1 [ 401.078187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 401.098656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 1 [ 401.118283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 401.135702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 1 [ 401.149047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 401.161741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 1 [ 401.179889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 401.190899] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 401.194648] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 401.277232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 401.289992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 401.301585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 401.313663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 401.325311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 401.337178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 401.352591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 401.370090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 401.381791] LNet: 57549:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 401.392446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 401.407639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 401.421505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 401.435624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 401.451966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 401.468588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 401.486557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 401.669338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 401.685769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 1 [ 401.704973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 401.718781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 1 [ 401.736163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 401.753645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 1 [ 401.770826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 401.787135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 1 [ 401.801985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 401.811493] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 401.813540] LNet: Skipped 2 previous similar messages [ 401.890924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 401.901424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 401.912704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 401.924320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 401.938749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 401.955126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 401.971959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 401.989037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 402.000990] LNet: 57670:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 402.010398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 402.023940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 402.035653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 402.047886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 402.059671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 402.072111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 402.084058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 402.272712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 402.292052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 1 [ 402.312714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 402.331900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 1 [ 402.348836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 402.365725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 1 [ 402.379087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 402.395028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 1 [ 402.410466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 402.502990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 402.517545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 402.532740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 402.547113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 402.563417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 402.579547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 402.596308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 402.612534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 402.629677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 402.646666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 402.663295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 402.681708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 402.698881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 402.716041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 402.732689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 402.925649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 402.944518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 1 [ 402.960451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 402.976232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 1 [ 402.993186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 403.006384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 1 [ 403.020204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 403.032752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 1 [ 403.046491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 403.054069] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 403.056593] LNet: Skipped 6 previous similar messages [ 403.128880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 403.139365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 403.149523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 403.159614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 403.170821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 403.182114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 403.193655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 403.205065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 403.212481] LNet: 57912:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 403.216558] LNet: 57912:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 403.223206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 403.232692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 403.242681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 403.252420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 403.264511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 403.276111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 403.287015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 403.465246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 403.483859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 1 [ 403.501617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 403.517643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 1 [ 403.530673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 403.542525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 1 [ 403.554779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 403.566642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 1 [ 403.579642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 403.663169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 403.675984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 403.690463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 403.705203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 403.717327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 403.729048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 403.740980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 403.753234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 403.768143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 403.783088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 403.793623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 403.808086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 403.824116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 403.835185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 403.845499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 404.021936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 404.039394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_error -r 1 [ 404.055316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 404.072125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_error -r 1 [ 404.088961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 404.103919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_error -r 1 [ 404.117620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 404.129979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_error -r 1 [ 404.142171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 404.220393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 404.237624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 404.252070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 404.266427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 404.280104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 404.293132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 404.309412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 404.324170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 404.338731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 404.353658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 404.369223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 404.383754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 404.399674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 404.415741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 404.431982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 406.533435] LNet: 58230:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.535905] LNetError: 58230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 406.538595] LNet: Removed LNI 192.168.203.16@tcp [ 407.589355] LNet: Removed LNI 192.168.203.16@tcp1 [ 407.753805] Key type .llcrypt unregistered [ 407.755606] Key type ._llcrypt unregistered [ 412.904212] Key type ._llcrypt registered [ 412.905327] Key type .llcrypt registered [ 413.368258] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:33:14 (1778355194) [ 413.649301] Key type .llcrypt unregistered [ 413.650817] Key type ._llcrypt unregistered [ 415.615934] Key type ._llcrypt registered [ 415.617838] Key type .llcrypt registered [ 415.688930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 415.695198] alg: No test for adler32 (adler32-zlib) [ 416.501001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 416.515400] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 419.259857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 419.518298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 419.535901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 419.551836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 419.561440] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 419.661418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 419.674127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 419.689722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 419.704251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 419.719277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 419.733444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 419.746181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 419.760062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 419.767524] LNet: 59475:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 419.774991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 419.786376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 419.797856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 419.809980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 419.821665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 419.837146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 419.851485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 420.061096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 420.076440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 420.088446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 420.095338] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 420.187407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 420.197111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 420.207307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 420.218104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 420.230429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 420.242216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 420.255131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 420.268796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 420.277346] LNet: 59584:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 420.286551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 420.298862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 420.312991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 420.329564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 420.346634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 420.363175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 420.377713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 420.575095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 420.594670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 420.614772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 420.730432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 420.744749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 420.755644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 420.768796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 420.783972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 420.800996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 420.818196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 420.835475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 420.853587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 420.870497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 420.887100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 420.901828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 420.917754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 420.930525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 420.942968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 421.047441] LNet: 59734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.051189] LNetError: 59734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 421.062480] LNet: Removed LNI 192.168.203.16@tcp [ 421.228972] Key type .llcrypt unregistered [ 421.230592] Key type ._llcrypt unregistered [ 426.814507] Key type ._llcrypt registered [ 426.816048] Key type .llcrypt registered [ 427.274913] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:33:28 (1778355208) [ 427.571380] Key type .llcrypt unregistered [ 427.572572] Key type ._llcrypt unregistered [ 429.463666] Key type ._llcrypt registered [ 429.464777] Key type .llcrypt registered [ 429.528469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 429.533234] alg: No test for adler32 (adler32-zlib) [ 430.339059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 430.351769] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 433.075887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 435.560844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.572383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 435.579344] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 436.035946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 436.052751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 1 [ 436.065935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 436.078461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 1 [ 436.091173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 436.104178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 1 [ 436.118641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 436.131359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 1 [ 436.144342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 436.153049] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 436.155498] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 436.233467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 436.245824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 436.259211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 436.275706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 436.288617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 436.302724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 436.315610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 436.326266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 436.333310] LNet: 61197:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 436.341339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 436.355355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 436.367930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 436.379754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 436.390462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 436.405536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 436.420691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 436.433414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 436.608705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 436.623421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_error -r 1 [ 436.640838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 436.654712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_error -r 1 [ 436.668966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 436.682857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_error -r 1 [ 436.698197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 436.714776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_error -r 1 [ 436.729311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 436.737160] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 436.739456] LNet: Skipped 2 previous similar messages [ 436.813614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 436.825786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 436.842290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 436.854462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 436.866716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 436.879879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 436.891709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 436.905131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 436.912348] LNet: 61320:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 436.920861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 436.933548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 436.947622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 436.958889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 436.973868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 436.987650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 436.998797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 437.009147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 437.181647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 437.194786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_timeout -r 1 [ 437.209414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 437.227279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_timeout -r 1 [ 437.243314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 437.258588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_timeout -r 1 [ 437.274687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 437.291100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_timeout -r 1 [ 437.307348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 437.389136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 437.399502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 437.409964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 437.420791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 437.433758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 437.445943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 437.456106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 437.466637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 437.476963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 437.487654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 437.499305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 437.511206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 437.524124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 437.538453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 437.551803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 437.567845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 439.684451] LNet: 61520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.688449] LNetError: 61520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 439.693247] LNet: Removed LNI 192.168.203.16@tcp [ 439.702333] LNet: Removed LNI 192.168.203.16@tcp1 [ 439.876292] Key type .llcrypt unregistered [ 439.878162] Key type ._llcrypt unregistered [ 445.247362] Key type ._llcrypt registered [ 445.248818] Key type .llcrypt registered [ 445.695096] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:33:46 (1778355226) [ 445.860288] Key type .llcrypt unregistered [ 445.861360] Key type ._llcrypt unregistered [ 447.681858] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 447.721836] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 447.723856] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 447.725454] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 449.964655] Key type ._llcrypt registered [ 449.966159] Key type .llcrypt registered [ 450.030768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 450.036235] alg: No test for adler32 (adler32-zlib) [ 450.850630] LNet: 62595:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 450.855303] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 450.868548] LNet: 62609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.870838] LNetError: 62609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 450.883493] LNet: Removed LNI 192.168.203.16@tcp [ 451.143436] Key type .llcrypt unregistered [ 451.145117] Key type ._llcrypt unregistered [ 453.315127] Key type ._llcrypt registered [ 453.316198] Key type .llcrypt registered [ 453.385709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 453.392451] alg: No test for adler32 (adler32-zlib) [ 454.213127] LNet: 62868:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 454.219148] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 454.221084] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 454.235569] LNet: 62882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.238107] LNetError: 62882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.240407] LNet: Removed LNI 192.168.203.16@tcp [ 454.249378] LNet: Removed LNI 10.1.2.3@tcp1 [ 454.498368] Key type .llcrypt unregistered [ 454.500153] Key type ._llcrypt unregistered [ 456.625456] Key type ._llcrypt registered [ 456.626958] Key type .llcrypt registered [ 456.694930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 456.702364] alg: No test for adler32 (adler32-zlib) [ 457.502377] LNet: 63140:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 457.507772] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 457.509058] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 457.522585] LNet: 63154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 457.526568] LNetError: 63154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 457.530024] LNet: Removed LNI 192.168.203.16@tcp [ 457.537302] LNet: Removed LNI 10.1.2.3@tcp [ 457.795381] Key type .llcrypt unregistered [ 457.797216] Key type ._llcrypt unregistered [ 459.784766] Key type ._llcrypt registered [ 459.785757] Key type .llcrypt registered [ 459.846618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 459.851761] alg: No test for adler32 (adler32-zlib) [ 460.661775] LNet: 63414:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 460.668847] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 460.670194] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 460.686657] LNet: 63428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.689989] LNetError: 63428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 460.693202] LNet: Removed LNI 192.168.203.16@tcp [ 460.698436] LNet: Removed LNI 10.1.2.3@tcp [ 460.989411] Key type .llcrypt unregistered [ 460.990748] Key type ._llcrypt unregistered [ 462.903137] Key type ._llcrypt registered [ 462.904123] Key type .llcrypt registered [ 462.964367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 462.970287] alg: No test for adler32 (adler32-zlib) [ 463.772691] LNet: 63691:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 463.776758] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 463.777895] LNetError: 63691:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 463.779989] LNetError: Error -100 starting up LNI tcp [ 463.788467] LNet: Removed LNI 192.168.203.16@tcp [ 464.016291] Key type .llcrypt unregistered [ 464.017359] Key type ._llcrypt unregistered [ 467.369389] Key type ._llcrypt registered [ 467.370622] Key type .llcrypt registered [ 467.799384] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:34:08 (1778355248) [ 468.068186] Key type .llcrypt unregistered [ 468.069539] Key type ._llcrypt unregistered [ 469.957405] Key type ._llcrypt registered [ 469.958418] Key type .llcrypt registered [ 470.014934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 470.020387] alg: No test for adler32 (adler32-zlib) [ 470.819582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 470.831077] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 473.562287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 473.756730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 473.772204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 473.787369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 473.796898] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 473.873829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 473.885848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 473.887084] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 473.895634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 473.907629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 473.918623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 473.928234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 473.939072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 473.948914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 473.955096] LNet: 64944:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 473.961371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 473.972483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 473.983468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 473.994109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 474.007600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 474.018895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 474.102756] LNet: 64983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.104820] LNetError: 64983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 474.113349] LNet: Removed LNI 192.168.203.16@tcp [ 474.242276] Key type .llcrypt unregistered [ 474.243764] Key type ._llcrypt unregistered [ 479.585332] Key type ._llcrypt registered [ 479.586384] Key type .llcrypt registered [ 479.644782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 479.650378] alg: No test for adler32 (adler32-zlib) [ 480.447579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 480.457979] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 483.302577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 486.052791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.067728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 486.078874] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 486.542767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 486.555268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e network_timeout -r 1 [ 486.567420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 486.579764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e network_timeout -r 1 [ 486.592655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 486.604443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e network_timeout -r 1 [ 486.617393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 486.630174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e network_timeout -r 1 [ 486.641643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 486.648660] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 486.721295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 486.731738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 486.743552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 486.758624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 486.772185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 486.785756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 486.800765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 486.812585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 486.819670] LNet: 65873:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 486.826260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 486.836629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 486.849751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 486.863988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 486.875604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 486.887734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 488.930472] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 489.019951] LNet: 65947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.022223] LNetError: 65947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 489.024880] LNet: Removed LNI 192.168.203.16@tcp [ 489.031364] LNet: Removed LNI 192.168.203.16@tcp1 [ 489.163355] Key type .llcrypt unregistered [ 489.164660] Key type ._llcrypt unregistered [ 494.190525] Key type ._llcrypt registered [ 494.191498] Key type .llcrypt registered [ 494.617943] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:34:35 (1778355275) [ 494.731881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 494.736937] alg: No test for adler32 (adler32-zlib) [ 495.541664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.561961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 495.571389] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 495.840829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 495.849039] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 496.152473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 496.182260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 496.203442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 496.220197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 496.236464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 496.252535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 496.268399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 496.277337] LNetError: 67044:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 496.607421] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 496.607502] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 498.655335] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 499.456658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 502.751440] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 503.666071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 503.675974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 503.693957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.700909] LNet: 67161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.703408] LNetError: 67161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 503.705538] LNet: Removed LNI 192.168.203.16@tcp [ 503.713234] LNet: Removed LNI 192.168.203.16@tcp1 [ 503.719891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.737228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 503.745394] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 503.974184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 503.980913] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 504.261737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 504.277167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 504.299151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 504.313779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 504.324579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 504.337089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 504.348723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 504.362227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 504.368903] LNetError: 67433:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 506.783311] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 506.787866] LNet: Skipped 1 previous similar message [ 507.564668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 510.879489] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 515.914297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 515.925580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 515.936146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 518.137638] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:34:59 (1778355299) [ 518.183744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.193500] LNet: 68158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.195884] LNetError: 68158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 518.197825] LNetError: 68158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 518.201123] LNet: Removed LNI 192.168.203.16@tcp [ 518.214592] LNet: Removed LNI 192.168.203.16@tcp1 [ 518.221913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.238937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 518.247433] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 518.479413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 518.487182] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 518.773133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 518.797392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 518.809557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 518.822798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 518.835776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 518.848686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 518.862497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 0 [ 518.899751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 519.263357] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.16@tcp [ 519.263423] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 519.270566] LNet: Skipped 1 previous similar message [ 534.441704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 534.506397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 534.520606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 534.535348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 500 [ 534.573905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 540.810269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 540.847680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.859884] LNet: 68658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.864302] LNetError: 68658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 540.869469] LNetError: 68658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 540.875083] LNet: Removed LNI 192.168.203.16@tcp [ 540.882365] LNet: Removed LNI 192.168.203.16@tcp1 [ 540.891431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.921282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 540.938153] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 541.200766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 541.211512] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 541.527895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 541.544408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 541.569122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 541.581630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 541.593398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 541.606087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 541.617586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 541.628562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 0 [ 542.943468] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 542.948355] LNet: Skipped 3 previous similar messages [ 543.802082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 553.263208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 553.278640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 553.291402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 555.647024] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:35:36 (1778355336) [ 555.795177] LNet: 69713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.797915] LNetError: 69713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 555.800195] LNetError: 69713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 555.803233] LNet: Removed LNI 192.168.203.16@tcp [ 555.810450] LNet: Removed LNI 192.168.203.16@tcp1 [ 556.030811] Key type .llcrypt unregistered [ 556.033409] Key type ._llcrypt unregistered [ 558.024892] Key type ._llcrypt registered [ 558.025866] Key type .llcrypt registered [ 558.084620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 558.090564] alg: No test for adler32 (adler32-zlib) [ 558.901528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 558.912702] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 562.474598] Lustre: DEBUG MARKER: Initial discovery [ 562.511547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 563.366431] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 563.383138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 563.404324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 563.416853] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 563.426990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 563.445518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.116@tcp -r 1 -m GET -l 3 [ 563.460836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.116@tcp -r 1 -m GET -e local_error [ 563.474316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.116@tcp -r 1 -m PUT -l 6 [ 563.987020] Lustre: DEBUG MARKER: Force oleg316-client.virtnet to discover 192.168.203.116@tcp (in background) [ 564.007407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 564.063456] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 564.516608] Lustre: DEBUG MARKER: Force oleg316-server to discover 192.168.203.16@tcp [ 565.350247] Lustre: DEBUG MARKER: Wait for 70466 [ 567.071394] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 567.774317] Lustre: DEBUG MARKER: Finished wait on 70466 [ 567.799504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 567.823994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 567.947155] LNet: 70922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.950437] LNetError: 70922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 567.963438] LNet: Removed LNI 192.168.203.16@tcp [ 568.134612] Key type .llcrypt unregistered [ 568.137137] Key type ._llcrypt unregistered [ 573.713787] Key type ._llcrypt registered [ 573.714782] Key type .llcrypt registered [ 574.181008] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:35:55 (1778355355) [ 574.368418] Key type .llcrypt unregistered [ 574.370150] Key type ._llcrypt unregistered [ 576.380231] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 576.421215] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 576.423042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 576.424655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 578.634279] Key type ._llcrypt registered [ 578.635360] Key type .llcrypt registered [ 578.706538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 578.713220] alg: No test for adler32 (adler32-zlib) [ 579.516565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.534366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 579.543737] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 579.782530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 579.790598] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 582.376748] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:36:03 (1778355363) [ 582.417111] LNet: 72988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 582.421181] LNetError: 72988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 582.424916] LNet: Removed LNI 192.168.203.16@tcp [ 582.440711] LNet: Removed LNI 10.1.2.3@tcp [ 582.668235] Key type .llcrypt unregistered [ 582.669275] Key type ._llcrypt unregistered [ 584.699505] Key type ._llcrypt registered [ 584.700905] Key type .llcrypt registered [ 584.762400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 584.768062] alg: No test for adler32 (adler32-zlib) [ 585.570327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.593184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.601661] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 585.877420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 585.885433] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 586.196163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.16@tcp [ 587.231146] LNetError: Unexpected error -22 connecting to 192.168.203.16@tcp at host 192.168.203.16:988 [ 588.408756] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:36:09 (1778355369) [ 588.475972] LNet: 74149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.479100] LNetError: 74149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.481114] LNet: Removed LNI 192.168.203.16@tcp [ 588.488500] LNet: Removed LNI 10.1.2.3@tcp [ 588.706285] Key type .llcrypt unregistered [ 588.707368] Key type ._llcrypt unregistered [ 590.581002] Key type ._llcrypt registered [ 590.582588] Key type .llcrypt registered [ 590.663423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 590.669485] alg: No test for adler32 (adler32-zlib) [ 591.478244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.495883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 591.503716] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 591.786731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 591.797178] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 592.160846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.16@tcp1 --nid 192.168.203.16@tcp2 [ 593.446419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp1 192.168.203.16@tcp2 [ 593.455233] LNetError: 74982:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.16@tcp1 to 192.168.203.16@tcp2 - Route table may be misconfigured [ 593.463037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp2 192.168.203.16@tcp1 [ 595.595076] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:36:16 (1778355376) [ 595.646502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.656159] LNet: 75553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.658914] LNetError: 75553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.703111] LNet: Removed LNI 192.168.203.16@tcp1 [ 596.704509] LNetError: 75553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.715716] LNet: Removed LNI 192.168.203.16@tcp2 [ 596.726789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.747263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.756275] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 596.991000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 596.998424] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 597.295521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 597.308998] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.16@tcp [ 597.321682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp1 192.168.203.16@tcp1 [ 597.336403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.16@tcp -d 192.168.203.16@tcp -e network_timeout [ 597.351152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -e network_timeout [ 597.369724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -e network_timeout [ 597.387289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -e network_timeout [ 597.406808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 597.416662] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 597.424627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 597.453849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 597.484336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 597.791100] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 599.499953] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:36:20 (1778355380) [ 599.540482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.549593] LNet: 76410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.552380] LNetError: 76410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 599.555280] LNet: Removed LNI 192.168.203.16@tcp [ 600.613406] LNet: Removed LNI 192.168.203.16@tcp1 [ 600.623037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.655346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 600.789741] LNet: 76452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.944240] Key type .llcrypt unregistered [ 600.945752] Key type ._llcrypt unregistered [ 604.594660] Key type ._llcrypt registered [ 604.595639] Key type .llcrypt registered [ 605.131449] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:36:26 (1778355386) [ 605.270687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 605.277796] alg: No test for adler32 (adler32-zlib) [ 606.076049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.554820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 607.066519] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:36:28 (1778355388) [ 607.096929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.108510] LNet: 77545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.120760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.140843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.150517] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 607.359262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 607.366301] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 607.633051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 607.645752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp1 [ 607.657620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp1 [ 609.752095] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:36:30 (1778355390) [ 610.330363] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg316-server [ 610.809140] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:36:32 (1778355392) [ 611.323610] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg316-server [ 611.836192] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:36:33 (1778355393) [ 612.472689] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg316-server [ 613.071192] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:36:34 (1778355394) [ 613.732982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg316-server [ 614.271715] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:36:35 (1778355395) [ 614.980987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg316-server [ 615.496127] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:36:36 (1778355396) [ 616.068589] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg316-server [ 616.590744] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:36:37 (1778355397) [ 617.175988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg316-server [ 617.707584] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:36:38 (1778355398) [ 618.310815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg316-server [ 618.823501] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:36:40 (1778355400) [ 619.381829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg316-server [ 619.923218] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:36:41 (1778355401) [ 619.982747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.989648] LNet: 81421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.991830] LNetError: 81421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.995280] LNet: Removed LNI 192.168.203.16@tcp [ 621.029707] LNet: Removed LNI 192.168.203.16@tcp1 [ 621.040280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.065750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 621.078370] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 621.317461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 621.343403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 621.398453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.406798] LNet: 81568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.409798] LNetError: 81568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.412905] LNetError: 81568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 621.422262] LNet: Removed LNI 192.168.203.16@tcp [ 621.429916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.449633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 621.458527] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 621.708651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 621.765681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.821621] LNet: Removed LNI 192.168.203.16@tcp [ 622.833089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.857102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 622.869670] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 623.136421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 623.176277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 623.251851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.260764] LNetError: 81860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.265089] LNetError: 81860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 623.273509] LNet: Removed LNI 192.168.203.16@tcp [ 623.281562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.308410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 623.320120] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 623.668251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 623.758047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.771881] LNet: 82005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.775670] LNet: 82005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 624.809289] LNet: Removed LNI 192.168.203.16@tcp [ 624.817837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.836266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 624.843959] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 625.079478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 625.105781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 626.231910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.247924] LNetError: 82162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.254682] LNetError: 82162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 626.265565] LNet: Removed LNI 192.168.203.16@tcp [ 626.279201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.309473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 626.321927] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 626.627495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 626.732415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.813274] LNet: Removed LNI 192.168.203.16@tcp [ 627.820455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.839397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 627.851072] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 628.146287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 628.178524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 629.329407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.343290] LNet: 82463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.347950] LNet: 82463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 630.373458] LNet: Removed LNI 192.168.203.16@tcp [ 630.386763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.419966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 630.437711] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 630.783158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 630.883855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.895399] LNetError: 82608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.898813] LNetError: 82608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 630.910382] LNet: Removed LNI 192.168.203.16@tcp [ 630.920805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.946245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 630.958426] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 631.288602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 631.333892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 631.454275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.485490] LNet: Removed LNI 192.168.203.16@tcp [ 632.499416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.534159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 632.553843] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 632.956467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 634.108668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.173322] LNet: Removed LNI 192.168.203.16@tcp [ 635.180188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.196780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.204928] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 635.435877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 635.467139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 635.542482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.581399] LNet: Removed LNI 192.168.203.16@tcp [ 636.588495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.606084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 636.615160] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 636.871083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 637.989957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.996803] LNet: 83211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.999070] LNet: 83211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 639.013394] LNet: Removed LNI 192.168.203.16@tcp [ 639.020493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.043603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 639.052031] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 639.288075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 639.311659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.320990] LNetError: 83344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.324523] LNetError: 83344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 639.334353] LNet: Removed LNI 192.168.203.16@tcp [ 639.343678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.361345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 639.369195] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 639.614062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 641.935849] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:37:03 (1778355423) [ 641.980598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.997075] LNet: Removed LNI 192.168.203.16@tcp [ 642.006169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.032252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 642.041374] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 642.464919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.477697] LNet: Removed LNI 192.168.203.16@tcp231 [ 642.489348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.523611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 642.537568] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 642.540927] LNet: 84193:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 642.700277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 642.717572] LNet: Removed LNI 192.168.203.16@tcp231 [ 642.723571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 642.733809] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 643.132597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.148605] LNet: Removed LNI 192.168.203.16@tcp231 [ 643.155797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.184246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 643.192962] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/0] [ 643.195021] LNet: 84345:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 643.321715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 643.337416] LNet: Removed LNI 192.168.203.16@tcp231 [ 643.343641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 643.351642] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/0] [ 643.782135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.800468] LNet: Removed LNI 192.168.203.16@tcp231 [ 643.810749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.835129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 643.843473] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/60] [ 643.984748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 644.002437] LNet: Removed LNI 192.168.203.16@tcp231 [ 644.009500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 644.019478] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/60] [ 644.435331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.449499] LNet: Removed LNI 192.168.203.16@tcp231 [ 644.457270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.482713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 644.493271] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 644.495553] LNet: 84648:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 644.497870] LNet: 84648:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 644.635660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 644.651868] LNet: Removed LNI 192.168.203.16@tcp231 [ 644.664272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 644.678375] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 647.283405] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:37:08 (1778355428) [ 647.324401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.342880] LNet: Removed LNI 192.168.203.16@tcp231 [ 647.354252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.385475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 647.395244] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 647.677284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 647.697355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 647.716748] LNet: Removed LNI 192.168.203.16@tcp232 [ 647.725829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 647.738392] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 648.068768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 648.091347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 648.112438] LNet: Removed LNI 192.168.203.16@tcp232 [ 648.123161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 648.139288] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 648.492310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 648.509925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 648.528572] LNet: Removed LNI 192.168.203.16@tcp232 [ 648.535652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 648.545383] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 648.813404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 648.832315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 648.849352] LNet: Removed LNI 192.168.203.16@tcp232 [ 648.855796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 650.955746] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:37:12 (1778355432) [ 651.287346] Key type .llcrypt unregistered [ 651.289674] Key type ._llcrypt unregistered [ 653.315441] Key type ._llcrypt registered [ 653.316522] Key type .llcrypt registered [ 653.376882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 653.382749] alg: No test for adler32 (adler32-zlib) [ 654.190070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 654.205857] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 657.158855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 659.867757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.879446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 659.886690] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 660.355449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 2 [ 660.367996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 2 [ 660.380901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 2 [ 660.393361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 2 [ 660.407745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_interrupt -r 2 [ 660.420481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 2 [ 660.435254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_interrupt -r 2 [ 660.448542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 2 [ 660.461949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 660.469537] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 660.471788] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 660.555670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 660.569027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 660.580542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 660.593543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 660.605864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 660.619267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 660.630545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 660.642224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 660.653394] LNet: 87145:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 660.662822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 660.675742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 660.688538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 660.703386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 660.715817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 660.727628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 660.911571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 2 [ 660.925640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 2 [ 660.938595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 2 [ 660.951374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 2 [ 660.965669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_dropped -r 2 [ 660.979732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 2 [ 660.991614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_dropped -r 2 [ 661.005150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 2 [ 661.017981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 661.024916] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 661.107479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 661.118374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 661.129280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 661.139753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 661.150029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 661.160062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 661.170913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 661.184439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 661.193527] LNet: 87265:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 661.200786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 661.211260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 661.225701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 661.243096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 661.257628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 661.268678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 661.445855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 2 [ 661.465407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 2 [ 661.481907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 2 [ 661.497314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 2 [ 661.511084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_aborted -r 2 [ 661.525463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 2 [ 661.540131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_aborted -r 2 [ 661.554696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 2 [ 661.569949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 661.652125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 661.662438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 661.673384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 661.684179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 661.696680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 661.707584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 661.720147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 661.734951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 661.750507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 661.762112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 661.772731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 661.783715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 661.794990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 661.805744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 661.998929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 2 [ 662.014771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 2 [ 662.029648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 2 [ 662.044215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 2 [ 662.064970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_no_route -r 2 [ 662.102127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 2 [ 662.130301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_no_route -r 2 [ 662.144281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 2 [ 662.157670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 662.166139] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 662.169104] LNet: Skipped 2 previous similar messages [ 662.248389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 662.259901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 662.271097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 662.282709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 662.295932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 662.309958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 662.324813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 662.338543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 662.345927] LNet: 87505:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 662.349266] LNet: 87505:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 662.356358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 662.368431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 662.382123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 662.397991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 662.412368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 662.426495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 662.635748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 2 [ 662.662142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 2 [ 662.674883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 2 [ 662.686272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 2 [ 662.698120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_timeout -r 2 [ 662.709339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 2 [ 662.720900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_timeout -r 2 [ 662.732405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 2 [ 662.743499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 662.833367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 662.847875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 662.860558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 662.870098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 662.880524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 662.889990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 662.900210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 662.910623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 662.920432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 662.930815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 662.940947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 662.952457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 662.962055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 662.973254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 663.150676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 2 [ 663.163345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 2 [ 663.178677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 2 [ 663.192742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 2 [ 663.207194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_dropped -r 2 [ 663.220261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 2 [ 663.235329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_dropped -r 2 [ 663.253288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 2 [ 663.269084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 663.280576] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 663.355919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 663.366302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 663.376373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 663.386256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 663.396592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 663.407465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 663.419131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 663.429492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 663.440637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 663.450687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 663.460208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 663.471893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 663.484274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 663.496255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 665.624909] LNet: 87819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.627796] LNetError: 87819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.630855] LNet: Removed LNI 192.168.203.16@tcp [ 665.638409] LNet: Removed LNI 192.168.203.16@tcp1 [ 665.814979] Key type .llcrypt unregistered [ 665.817474] Key type ._llcrypt unregistered [ 671.047867] Key type ._llcrypt registered [ 671.049430] Key type .llcrypt registered [ 671.586663] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:37:32 (1778355452) [ 671.728100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 671.733536] alg: No test for adler32 (adler32-zlib) [ 672.542403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.560619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 672.568362] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 672.889160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 672.902302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 672.911832] LNetError: 88795:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 672.949089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.16@tcp [ 672.988590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 675.234634] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:37:36 (1778355456) [ 675.283948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.296663] LNet: 89381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.299733] LNetError: 89381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 675.308970] LNet: Removed LNI 192.168.203.16@tcp [ 675.323036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.350818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 675.359103] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 675.643502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 675.652757] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 676.055409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.16@tcp [ 676.064444] LNetError: 89651:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 676.383422] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 676.384400] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 677.162338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.16@tcp [ 681.492112] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:37:42 (1778355462) [ 682.222488] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg316-server [ 682.941219] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:37:43 (1778355463) [ 683.722888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg316-server [ 684.352527] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:37:45 (1778355465) [ 685.116459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg316-server [ 685.777819] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:37:46 (1778355466) [ 685.824384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.834224] LNet: 91283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.836575] LNetError: 91283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.838406] LNet: Removed LNI 192.168.203.16@tcp [ 685.846968] LNet: Removed LNI 192.168.203.16@tcp1 [ 685.859668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.884151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.899379] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 686.249754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 686.292192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 686.328213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 686.376369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 686.406510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 686.428062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 689.342097] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:37:50 (1778355470) [ 689.367589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.377542] LNet: 92007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.381223] LNetError: 92007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.383949] LNetError: 92007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 689.396087] LNet: Removed LNI 192.168.203.16@tcp [ 689.406179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.428609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 689.439893] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 692.278635] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:37:53 (1778355473) [ 692.321960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.333632] LNet: 92710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.336475] LNetError: 92710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.345050] LNet: Removed LNI 192.168.203.16@tcp [ 692.355735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.378553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 692.387698] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 695.259912] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:37:56 (1778355476) [ 695.963351] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 696.785947] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:37:57 (1778355477) [ 696.989543] LNet: 93748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.992737] LNetError: 93748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.004154] LNet: Removed LNI 192.168.203.16@tcp [ 697.335398] Key type .llcrypt unregistered [ 697.337791] Key type ._llcrypt unregistered [ 699.897609] Key type ._llcrypt registered [ 699.898835] Key type .llcrypt registered [ 699.966968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 699.972368] alg: No test for adler32 (adler32-zlib) [ 700.811409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 700.828307] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 704.188238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 706.009752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.116@tcp [ 706.100459] LNet: 94164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.103150] LNetError: 94164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.110286] LNet: Removed LNI 192.168.203.16@tcp [ 706.246407] Key type .llcrypt unregistered [ 706.247457] Key type ._llcrypt unregistered [ 710.312475] Key type ._llcrypt registered [ 710.313674] Key type .llcrypt registered [ 710.922663] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 711.601040] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 712.295764] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:38:13 (1778355493) [ 712.453409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 712.460115] alg: No test for adler32 (adler32-zlib) [ 713.266486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.301763] LNet: 95249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.588500] Key type .llcrypt unregistered [ 713.590354] Key type ._llcrypt unregistered [ 715.793519] Key type ._llcrypt registered [ 715.794707] Key type .llcrypt registered [ 715.843457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 715.848603] alg: No test for adler32 (adler32-zlib) [ 716.663079] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 716.665995] LNet: 95504:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 716.670470] LNet: 95504:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 716.678809] LNetError: 95516:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.18@tcp is being used as a gateway but routing feature is not turned on [ 716.685296] LNetError: 95516:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.18@tcp has gone from up to down [ 716.850152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.327154] LNetError: 95516:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.20@tcp has gone from up to down [ 735.334889] LNet: 95562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.340370] LNetError: 95562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.350615] LNet: Removed LNI 192.168.203.16@tcp [ 735.365313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.406150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 735.422679] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 735.426399] LNet: 95572:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 735.432808] LNet: 95572:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 735.438701] LNetError: 95572:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.203.17@tcp is a gateway. Can not delete it [ 735.445377] LNetError: 95572:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.203.17@tcp is a router. Can not be deleted [ 735.451236] LNet: 95572:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 738.174219] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:38:39 (1778355519) [ 738.527622] LNetError: 95583:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.17@tcp has gone from up to down [ 738.532854] LNetError: 95583:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 738.539198] LNetError: 95569:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.18@tcp is being used as a gateway but routing feature is not turned on [ 739.065990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg316-server [ 739.843493] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:38:40 (1778355520) [ 740.726832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg316-server [ 741.455280] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:38:42 (1778355522) [ 742.113317] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 743.775256] Key type lgssc registered [ 744.272211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 752.532746] Key type lgssc unregistered [ 752.680714] LNet: 97989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.683423] LNetError: 97989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.690567] LNet: Removed LNI 192.168.203.16@tcp [ 752.882639] Key type .llcrypt unregistered [ 752.884051] Key type ._llcrypt unregistered [ 758.132512] Key type ._llcrypt registered [ 758.133852] Key type .llcrypt registered [ 758.655717] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:38:59 (1778355539) [ 758.784918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 758.789845] alg: No test for adler32 (adler32-zlib) [ 759.591302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.614892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 759.631769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 759.647598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 759.664060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 762.023857] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:39:03 (1778355543) [ 762.068233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.083908] LNet: 99387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.098816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.128769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 762.138906] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 762.486704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.496507] LNetError: 99536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.503323] LNet: Removed LNI 192.168.203.16@tcp [ 762.509626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.529121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 762.540158] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 762.704508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.713795] LNet: 99564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.715696] LNet: 99564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 762.721289] LNet: Removed LNI 192.168.203.16@tcp [ 762.727491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.746824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 762.759478] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 765.087755] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:39:06 (1778355546) [ 765.137494] LNet: 100152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.142712] LNetError: 100152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.146082] LNetError: 100152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 765.155994] LNet: Removed LNI 192.168.203.16@tcp [ 765.378585] Key type .llcrypt unregistered [ 765.380456] Key type ._llcrypt unregistered [ 767.458263] Key type ._llcrypt registered [ 767.459357] Key type .llcrypt registered [ 767.526158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 767.532441] alg: No test for adler32 (adler32-zlib) [ 768.338423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.362373] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 768.363672] LNet: *** cfs_fail_loc=e003, val=100*** [ 768.613000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 768.622376] Ping reply received with mapping: [ 768.624091] found metadata reply NI[192.168.203.16@tcp] [ 768.625814] 10 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 768.627688] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 768.629505] 5a 5a 5a 5a 5a 5a 5a 5a [ 768.674770] LNet: 100552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.676922] LNetError: 100552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.683474] LNet: Removed LNI 192.168.203.16@tcp [ 768.895298] Key type .llcrypt unregistered [ 768.896364] Key type ._llcrypt unregistered [ 772.548563] Key type ._llcrypt registered [ 772.549734] Key type .llcrypt registered [ 773.059773] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:39:14 (1778355554) [ 773.229275] Key type .llcrypt unregistered [ 773.231882] Key type ._llcrypt unregistered [ 775.100898] Key type ._llcrypt registered [ 775.102199] Key type .llcrypt registered [ 775.168826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 775.176309] alg: No test for adler32 (adler32-zlib) [ 776.056645] LNetError: 101596:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 776.196287] Key type .llcrypt unregistered [ 776.197468] Key type ._llcrypt unregistered [ 778.268538] Key type ._llcrypt registered [ 778.270313] Key type .llcrypt registered [ 778.765495] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:39:19 (1778355559) [ 778.970301] Key type .llcrypt unregistered [ 778.971600] Key type ._llcrypt unregistered [ 781.099841] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 781.101666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 783.316969] Key type ._llcrypt registered [ 783.318587] Key type .llcrypt registered [ 783.408575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 783.414788] alg: No test for adler32 (adler32-zlib) [ 784.235620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.259242] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 788.116244] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:39:29 (1778355569) [ 788.158324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.170677] LNet: 103368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.173330] LNetError: 103368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 788.184690] LNet: Removed LNI 10.1.2.3@tcp [ 788.198605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.476534] Key type .llcrypt unregistered [ 788.478286] Key type ._llcrypt unregistered [ 790.429448] Key type ._llcrypt registered [ 790.430804] Key type .llcrypt registered [ 790.485154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.490829] alg: No test for adler32 (adler32-zlib) [ 791.300544] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 791.325644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 791.389749] LNet: 103661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.393398] LNetError: 103661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.402447] LNet: Removed LNI 192.168.203.16@tcp [ 791.688721] Key type .llcrypt unregistered [ 791.690302] Key type ._llcrypt unregistered [ 795.365643] Key type ._llcrypt registered [ 795.366633] Key type .llcrypt registered [ 795.812353] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:39:36 (1778355576) [ 795.999321] Key type .llcrypt unregistered [ 796.000506] Key type ._llcrypt unregistered [ 797.869955] Key type ._llcrypt registered [ 797.871577] Key type .llcrypt registered [ 797.936164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 797.942136] alg: No test for adler32 (adler32-zlib) [ 800.739615] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:39:41 (1778355581) [ 800.786516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.800438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.822178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 800.830612] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 801.066167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.077920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 801.095509] LNet: Removed LNI 192.168.203.16@tcp [ 801.186159] LNet: 105402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.321310] Key type .llcrypt unregistered [ 801.322494] Key type ._llcrypt unregistered [ 805.041527] Key type ._llcrypt registered [ 805.042689] Key type .llcrypt registered [ 805.465628] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:39:46 (1778355586) [ 805.584185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.589220] alg: No test for adler32 (adler32-zlib) [ 806.389507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.408308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 806.416854] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 806.652773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 806.667127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.16@tcp [ 806.674673] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 806.676263] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 806.678010] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 806, 0, 0 [ 806.680539] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 806.681899] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 806.684504] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 806.685789] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 806, 0, 0 [ 806.687515] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 806.688484] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 806.689824] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 806.691132] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 806, 0, 856 [ 806.692839] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 806.693823] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 806.695188] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 806.696365] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 806, 0, 856 [ 806.698111] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 808.741293] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:39:49 (1778355589) [ 808.882314] LNet: 106939:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.884794] LNetError: 106939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.892653] LNet: Removed LNI 192.168.203.16@tcp [ 809.106301] Key type .llcrypt unregistered [ 809.107434] Key type ._llcrypt unregistered [ 810.906664] Key type ._llcrypt registered [ 810.908340] Key type .llcrypt registered [ 810.968295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.972929] alg: No test for adler32 (adler32-zlib) [ 811.769783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 811.780073] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 814.956109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 817.626205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.641215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 817.653220] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 817.988222] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 818.038406] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 818.040858] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 818.042608] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 820.197248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 820.204890] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 823.670859] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:40:04 (1778355604) [ 823.715335] LNet: 108431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.718607] LNetError: 108431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.722749] LNet: Removed LNI 192.168.203.16@tcp [ 824.735131] LNet: Removed LNI 192.168.203.16@tcp1 [ 824.736334] LNetError: 108431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 824.738453] LNetError: 108431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 824.746264] LNet: Removed LNI 10.1.2.3@tcp99 [ 824.955600] Key type .llcrypt unregistered [ 824.956646] Key type ._llcrypt unregistered [ 827.154583] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 827.156182] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 829.353980] Key type ._llcrypt registered [ 829.355091] Key type .llcrypt registered [ 829.429324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 829.436621] alg: No test for adler32 (adler32-zlib) [ 830.259102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.280747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 830.293507] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 830.614514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 830.624567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 831.414951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 831.433816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 831.448838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 831.455803] LNetError: 109179:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 831.468475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 831.484374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 831.618959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 831.648576] LNet: 109199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.652374] LNetError: 109199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.655619] LNet: Removed LNI 192.168.203.16@tcp [ 831.663409] LNet: Removed LNI 10.1.2.3@tcp [ 831.886298] Key type .llcrypt unregistered [ 831.887735] Key type ._llcrypt unregistered [ 835.589383] Key type ._llcrypt registered [ 835.590402] Key type .llcrypt registered [ 836.040940] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:40:17 (1778355617) [ 836.244305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 836.249408] alg: No test for adler32 (adler32-zlib) [ 837.047914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.065801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 837.073371] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 837.357270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 837.384317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg316-client@tcp [ 840.030741] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:40:21 (1778355621) [ 840.075325] LNet: 110723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.078509] LNetError: 110723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.088156] LNet: Removed LNI 192.168.203.16@tcp [ 840.369762] Key type .llcrypt unregistered [ 840.372664] Key type ._llcrypt unregistered [ 842.414672] Key type ._llcrypt registered [ 842.416144] Key type .llcrypt registered [ 842.471807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.477842] alg: No test for adler32 (adler32-zlib) [ 843.312335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 843.330155] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 843.352969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 845.621118] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:40:26 (1778355626) [ 846.195464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 846.866119] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:40:28 (1778355628) [ 847.414919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 848.139513] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:40:29 (1778355629) [ 848.202479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.214438] LNet: 112144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.218137] LNetError: 112144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.234786] LNet: Removed LNI 192.168.203.16@tcp [ 848.246223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.271317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.283692] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 848.611386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 848.635045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 851.133712] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:40:32 (1778355632) [ 851.178144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.189662] LNet: 112850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.192559] LNetError: 112850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.199694] LNet: Removed LNI 192.168.203.16@tcp [ 851.210886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.231591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 851.244170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 851.538300] Key type .llcrypt unregistered [ 851.540214] Key type ._llcrypt unregistered [ 855.220191] Key type ._llcrypt registered [ 855.222673] Key type .llcrypt registered [ 855.836525] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:40:36 (1778355636) [ 855.973565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.979258] alg: No test for adler32 (adler32-zlib) [ 856.788505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.807923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.816801] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 857.087488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 857.112589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 859.459907] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:40:40 (1778355640) [ 859.490882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.501033] LNet: 114403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.505778] LNetError: 114403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.519335] LNet: Removed LNI 192.168.203.16@tcp [ 859.531352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.550575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 859.561662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 861.971772] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:40:43 (1778355643) [ 862.033384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.044723] LNet: 114987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.059675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.093053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 862.200465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 862.210677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 862.237947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 862.367034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 862.378506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 862.404202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.416783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.439393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 862.553434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 862.566122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 862.590355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 862.707720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 862.724229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 865.611134] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:40:46 (1778355646) [ 865.687874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.700489] LNet: 115617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.704742] LNet: 115617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 865.727297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.755098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.772378] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 866.098218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 866.107903] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 866.541103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 866.577480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 866.707744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.720367] LNetError: 115900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.724038] LNet: Removed LNI 192.168.203.16@tcp [ 866.733743] LNet: Removed LNI 192.168.203.16@tcp2 [ 866.746434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.784449] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 866.788720] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 866.792533] LNet: 115906:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 866.991465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 867.120986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.191174] LNet: Removed LNI 192.168.203.16@tcp [ 868.194502] LNetError: 115943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.200079] LNetError: 115943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 868.213692] LNet: Removed LNI 192.168.203.16@tcp2 [ 868.230164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.259585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 868.276603] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 868.679472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 868.692648] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 869.105981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 869.126526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 869.162593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 869.295899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.312766] LNet: 116227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.317644] LNet: 116227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 870.367281] LNet: Removed LNI 192.168.203.16@tcp [ 870.370366] LNetError: 116227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.375801] LNetError: 116227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 870.387546] LNet: Removed LNI 192.168.203.16@tcp2 [ 870.404874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.442878] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 870.447459] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 870.451178] LNet: 116233:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 870.641049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 870.748955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.760903] LNet: Removed LNI 192.168.203.16@tcp [ 871.781370] LNet: Removed LNI 192.168.203.16@tcp2 [ 871.792478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.820478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 871.832894] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 872.246349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 872.260810] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 872.772481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 872.810137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 872.945307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.015134] LNet: Removed LNI 192.168.203.16@tcp [ 874.023538] LNet: Removed LNI 192.168.203.16@tcp2 [ 874.034631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.069178] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 874.073357] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 874.077711] LNet: 116556:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 874.264866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 874.359724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.368469] LNet: 116593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.371289] LNet: 116593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 874.374487] LNetError: 116593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.377457] LNetError: 116593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 874.380449] LNet: Removed LNI 192.168.203.16@tcp [ 874.387402] LNet: Removed LNI 192.168.203.16@tcp2 [ 874.396674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.418932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 874.429415] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 874.707446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 874.717320] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 875.069834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 875.086228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 875.109874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 875.193181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 876.255113] LNet: Removed LNI 192.168.203.16@tcp [ 877.285629] LNet: Removed LNI 192.168.203.16@tcp2 [ 877.300726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.338515] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 877.342735] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 877.346465] LNet: 116883:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 877.512660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 877.597203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.610759] LNet: Removed LNI 192.168.203.16@tcp [ 878.633528] LNet: Removed LNI 192.168.203.16@tcp2 [ 878.647607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.680181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 878.692277] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 878.978243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 878.988982] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 879.305421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 879.320330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 879.344309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 879.472990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.543208] LNet: Removed LNI 192.168.203.16@tcp [ 880.551423] LNet: Removed LNI 192.168.203.16@tcp2 [ 880.564448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.602370] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 880.606124] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 880.768729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 880.880421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.951184] LNet: Removed LNI 192.168.203.16@tcp [ 881.959425] LNet: Removed LNI 192.168.203.16@tcp2 [ 881.971791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.000679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.012847] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 882.264549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 882.272951] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 882.605930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 882.622705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 882.635639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 882.658801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 882.785767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.792848] LNet: 117576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.794789] LNet: 117576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 882.797743] LNetError: 117576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.800277] LNetError: 117576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 882.802961] LNet: Removed LNI 192.168.203.16@tcp [ 883.813449] LNet: Removed LNI 192.168.203.16@tcp2 [ 883.826751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.858089] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 883.861729] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 883.864516] LNet: 117582:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 883.868012] LNet: 117582:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 884.034357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 886.373688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 887.028223] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:41:08 (1778355668) [ 887.086451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.099098] LNet: Removed LNI 192.168.203.16@tcp [ 888.166075] LNet: Removed LNI 192.168.203.16@tcp2 [ 888.178276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.278897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.289496] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 888.524292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 888.532743] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 888.877638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.886791] LNet: Removed LNI 192.168.203.16@tcp [ 888.893405] LNet: Removed LNI 10.1.2.3@tcp [ 888.900479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.924086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 888.933771] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 888.936371] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 889.071424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 889.080889] LNet: Removed LNI 192.168.203.16@tcp [ 889.090308] LNet: Removed LNI 10.1.2.3@tcp [ 891.785805] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:41:12 (1778355672) [ 891.915038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.935354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.982530] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 892.024090] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 892.026014] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 892.027937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 894.194264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.203846] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 894.480609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 894.491609] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.313625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 895.334677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 895.372655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.16@tcp -m PUT -e local_timeout -r 1 [ 895.408934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 895.440087] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 895.443216] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 895.579979] LNetError: 119280:0:(lib-move.c:4969:lnet_parse()) 192.168.203.16@tcp, src 192.168.203.16@tcp: Dropping PUT (error -108 looking up sender) [ 897.631096] LNet: Removed LNI 192.168.203.16@tcp [ 897.639359] LNet: Removed LNI 10.1.2.3@tcp [ 897.788908] Key type .llcrypt unregistered [ 897.790231] Key type ._llcrypt unregistered [ 901.533862] Key type ._llcrypt registered [ 901.535020] Key type .llcrypt registered [ 901.968678] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:41:23 (1778355683) [ 902.107467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 902.112688] alg: No test for adler32 (adler32-zlib) [ 902.911532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.039879] LNet: 120591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.260237] Key type .llcrypt unregistered [ 903.261214] Key type ._llcrypt unregistered [ 905.312550] Key type ._llcrypt registered [ 905.313674] Key type .llcrypt registered [ 905.375032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 905.380245] alg: No test for adler32 (adler32-zlib) [ 906.182281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 906.192631] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 909.087797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 909.148248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 909.333449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.116@tcp [ 909.411124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 909.421749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 909.431953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 909.443407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 909.456407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 909.468947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 909.480196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 909.490899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 909.497844] LNet: 121039:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 909.504532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 909.516998] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 909.531346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 909.545542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 909.557796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 909.569982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 909.794643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 909.878979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 909.893259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 909.908437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 909.921047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 909.933212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 909.944550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 909.956055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 909.967264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 909.978693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 909.990343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 910.002825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 910.013747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 910.023636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 910.034240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 910.245913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.116@tcp [ 910.303195] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 910.329742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 910.340031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 910.350080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 910.360802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 910.372726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 910.385728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 910.396891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 910.408894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 910.415738] LNet: 121274:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 910.418050] LNet: 121274:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 910.424515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 910.436218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 910.447524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 910.458512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 910.471268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 910.485920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 910.685250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 910.762106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 910.771898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 910.782374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 910.792646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 910.803637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 910.814401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 910.825105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 910.837048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 910.848504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 910.859731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 910.870729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 910.881352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 910.891991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 910.902834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 910.996519] LNet: 121437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.998800] LNetError: 121437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.012338] LNet: Removed LNI 192.168.203.16@tcp [ 911.152565] Key type .llcrypt unregistered [ 911.153993] Key type ._llcrypt unregistered [ 916.698653] Key type ._llcrypt registered [ 916.699787] Key type .llcrypt registered [ 917.270854] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:41:38 (1778355698) [ 917.442615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 917.449370] alg: No test for adler32 (adler32-zlib) [ 918.250363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.369112] LNet: 122322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.603052] Key type .llcrypt unregistered [ 918.605192] Key type ._llcrypt unregistered [ 920.647086] Key type ._llcrypt registered [ 920.648066] Key type .llcrypt registered [ 920.708846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 920.714210] alg: No test for adler32 (adler32-zlib) [ 921.521726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 921.533386] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 924.618510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 927.843440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.856443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 927.862850] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 928.170851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 928.383542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.116@tcp [ 928.475925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 928.489662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 928.501452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 928.514361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 928.525931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 928.538726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 928.551910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 928.563082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 928.569079] LNet: 122972:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 928.575618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 928.586879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 928.598817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 928.609504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 928.620470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 928.631364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 928.871467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 928.959287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 928.970684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 928.981740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 928.992552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 929.006502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 929.018753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 929.032749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 929.046596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 929.060295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 929.073627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 929.085808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 929.097716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 929.111687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 929.126311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 929.366799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.116@tcp [ 929.457336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 929.468996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 929.482544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 929.494787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 929.508590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 929.522049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 929.533488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 929.551601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 929.561777] LNet: 123206:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 929.563982] LNet: 123206:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 929.572581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 929.586814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 929.599381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 929.611618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 929.624312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 929.642762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 929.920552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 930.023063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 930.036606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 930.049782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 930.062192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 930.073884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 930.085298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 930.097053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 930.109369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 930.120891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 930.132857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 930.144561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 930.156640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 930.168706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 930.180942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 932.356080] LNet: 123404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.358244] LNetError: 123404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.360734] LNet: Removed LNI 192.168.203.16@tcp [ 932.371347] LNet: Removed LNI 192.168.203.16@tcp1 [ 932.500593] Key type .llcrypt unregistered [ 932.501739] Key type ._llcrypt unregistered [ 938.147374] Key type ._llcrypt registered [ 938.149205] Key type .llcrypt registered [ 938.574553] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:41:59 (1778355719) [ 941.477465] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:42:02 (1778355722) [ 944.179939] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:42:05 (1778355725) [ 946.847862] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:42:08 (1778355728) [ 949.478316] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:42:10 (1778355730) [ 952.152089] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:42:13 (1778355733) [ 954.840144] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:42:16 (1778355736) [ 957.343503] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:42:18 (1778355738) [ 960.033461] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:42:21 (1778355741) [ 962.589989] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:42:23 (1778355743) [ 965.276687] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:42:26 (1778355746) [ 967.912589] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:42:29 (1778355749) [ 970.197537] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:42:31 (1778355751) [ 972.535968] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:42:33 (1778355753) [ 975.065289] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:42:36 (1778355756) [ 977.658120] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:42:38 (1778355758) [ 980.712913] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:42:41 (1778355761) [ 983.240244] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:42:44 (1778355764) [ 983.432163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 983.440088] alg: No test for adler32 (adler32-zlib) [ 984.244714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.288968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 984.296512] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 984.675379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.682809] LNet: 134101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.685347] LNetError: 134101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.700303] LNet: Removed LNI 192.168.203.16@tcp [ 984.707430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.728638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 984.741563] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 987.139288] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:42:48 (1778355768) [ 989.434662] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:42:50 (1778355770) [ 991.805821] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:42:52 (1778355772) [ 994.773473] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:42:55 (1778355775) [ 997.624502] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:42:58 (1778355778) [ 1000.557453] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:43:01 (1778355781) [ 1003.851856] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:43:04 (1778355784) [ 1007.149449] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:43:08 (1778355788) [ 1010.393738] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:43:11 (1778355791) [ 1012.664168] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:43:13 (1778355793) [ 1014.725222] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:43:15 (1778355795) [ 1016.954406] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:43:18 (1778355798) [ 1019.265773] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 910 sec =========== 15:43:20 (1778355800) [ 1019.391528] LNet: 141559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.393969] LNetError: 141559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.401417] LNet: Removed LNI 192.168.203.16@tcp [ 1019.642467] Key type .llcrypt unregistered [ 1019.643618] Key type ._llcrypt unregistered [ 1023.886349] Key type ._llcrypt registered [ 1023.887407] Key type .llcrypt registered [ 1024.056322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1024.060916] alg: No test for adler32 (adler32-zlib) [ 1024.920323] Lustre: Lustre: Build Version: 2.17.52_4_g786747d [ 1025.007657] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1026.583161] Key type lgssc registered [ 1027.011253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1051.721950] Lustre: Mounted lustre-client [ 1054.673626] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1062.408349] LustreError: 144004:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8c89180c8000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1062.428846] Lustre: Unmounted lustre-client [ 1076.062559] Key type lgssc unregistered [ 1076.216923] LNet: 144484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.221493] LNetError: 144484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.231932] LNet: Removed LNI 192.168.203.16@tcp [ 1076.575247] Key type .llcrypt unregistered [ 1076.577393] Key type ._llcrypt unregistered