[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 464176070 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004019] kvm-guest: setup PV IPIs [ 0.007717] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009017] pid_max: default: 32768 minimum: 301 [ 0.010176] LSM: Security Framework initializing [ 0.011064] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013085] *** VALIDATE selinux *** [ 0.021777] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026559] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027170] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029142] *** VALIDATE tmpfs *** [ 0.031510] *** VALIDATE proc *** [ 0.033186] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035309] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036177] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038036] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.041959] debug: unmapping init [mem 0xffffffff8c459000-0xffffffff8c460fff] [ 0.043250] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044775] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045028] ... version: 2 [ 0.045976] ... bit width: 48 [ 0.046017] ... generic registers: 4 [ 0.046906] ... value mask: 0000ffffffffffff [ 0.047022] ... max period: 00007fffffffffff [ 0.048017] ... fixed-purpose events: 3 [ 0.048950] ... event mask: 000000070000000f [ 0.049296] rcu: Hierarchical SRCU implementation. [ 0.051619] smp: Bringing up secondary CPUs ... [ 0.052576] x86: Booting SMP configuration: [ 0.053030] .... node #0, CPUs: #1 #2 #3 [ 0.058151] smp: Brought up 1 node, 4 CPUs [ 0.059725] smpboot: Max logical packages: 1 [ 0.060010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.140361] node 0 deferred pages initialised in 78ms [ 0.143383] devtmpfs: initialized [ 0.144297] x86/mm: Memory block size: 128MB [ 0.147820] gcov: version magic: 0x41383552 [ 0.150281] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152151] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154370] pinctrl core: initialized pinctrl subsystem [ 0.156232] [ 0.156599] ************************************************************* [ 0.158015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159009] ** ** [ 0.161014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163012] ** ** [ 0.165018] ** This means that this kernel is built to expose internal ** [ 0.166011] ** IOMMU data structures, which may compromise security on ** [ 0.168015] ** your system. ** [ 0.169011] ** ** [ 0.171011] ** If you see this message and you are not debugging the ** [ 0.173012] ** kernel, report this immediately to your vendor! ** [ 0.174011] ** ** [ 0.176011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177010] ************************************************************* [ 0.179755] NET: Registered protocol family 16 [ 0.180482] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186030] cpuidle: using governor menu [ 0.187391] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.189426] PCI: Using configuration type 1 for base access [ 0.190127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.199102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.201036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.203153] cryptd: max_cpu_qlen set to 1000 [ 0.206258] ACPI: Added _OSI(Module Device) [ 0.207019] ACPI: Added _OSI(Processor Device) [ 0.208019] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.209014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215650] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220585] ACPI: Interpreter enabled [ 0.222077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223013] ACPI: Using IOAPIC for interrupt routing [ 0.224133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227392] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.235456] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.241091] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.244330] acpiphp: Slot [2] registered [ 0.245118] acpiphp: Slot [5] registered [ 0.246105] acpiphp: Slot [6] registered [ 0.247086] acpiphp: Slot [3] registered [ 0.248047] acpiphp: Slot [4] registered [ 0.249128] acpiphp: Slot [7] registered [ 0.251095] acpiphp: Slot [8] registered [ 0.252152] acpiphp: Slot [9] registered [ 0.253094] acpiphp: Slot [10] registered [ 0.254000] acpiphp: Slot [11] registered [ 0.254000] acpiphp: Slot [12] registered [ 0.255087] acpiphp: Slot [13] registered [ 0.257091] acpiphp: Slot [14] registered [ 0.258094] acpiphp: Slot [15] registered [ 0.259108] acpiphp: Slot [16] registered [ 0.260082] acpiphp: Slot [17] registered [ 0.261097] acpiphp: Slot [18] registered [ 0.262108] acpiphp: Slot [19] registered [ 0.263089] acpiphp: Slot [20] registered [ 0.264071] acpiphp: Slot [21] registered [ 0.265099] acpiphp: Slot [22] registered [ 0.266106] acpiphp: Slot [23] registered [ 0.267102] acpiphp: Slot [24] registered [ 0.268084] acpiphp: Slot [25] registered [ 0.269069] acpiphp: Slot [26] registered [ 0.270095] acpiphp: Slot [27] registered [ 0.272064] acpiphp: Slot [28] registered [ 0.272945] acpiphp: Slot [29] registered [ 0.273066] acpiphp: Slot [30] registered [ 0.273855] acpiphp: Slot [31] registered [ 0.275093] PCI host bridge to bus 0000:00 [ 0.276030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.277013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.278016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.280021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.281024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.283025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.284187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.287134] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.290211] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.295679] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.298650] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.301015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.302010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.303012] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.305846] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.307777] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.309035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.311936] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.315028] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.324027] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.329015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.333466] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.344019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.363036] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.391029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.414339] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.420025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.427019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.440015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.448201] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.450387] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.452357] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.453251] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.454167] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.458114] iommu: Default domain type: Passthrough [ 0.459368] SCSI subsystem initialized [ 0.460089] ACPI: bus type USB registered [ 0.461030] usbcore: registered new interface driver usbfs [ 0.462000] usbcore: registered new interface driver hub [ 0.464089] usbcore: registered new device driver usb [ 0.465185] pps_core: LinuxPPS API ver. 1 registered [ 0.465988] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.468055] PTP clock support registered [ 0.470056] EDAC MC: Ver: 3.0.0 [ 0.471255] PCI: Using ACPI for IRQ routing [ 0.472802] NetLabel: Initializing [ 0.474017] NetLabel: domain hash size = 128 [ 0.475010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.476118] NetLabel: unlabeled traffic allowed by default [ 0.478143] vgaarb: loaded [ 0.479317] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.480011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.487000] clocksource: Switched to clocksource kvm-clock [ 0.603051] VFS: Disk quotas dquot_6.6.0 [ 0.605028] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.607367] *** VALIDATE ramfs *** [ 0.608501] *** VALIDATE hugetlbfs *** [ 0.609735] pnp: PnP ACPI init [ 0.611897] pnp: PnP ACPI: found 6 devices [ 0.631741] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.634382] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.636532] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.638297] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.639850] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.641550] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.643321] NET: Registered protocol family 2 [ 0.645217] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.649274] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.651958] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.656469] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.659485] TCP: Hash tables configured (established 65536 bind 65536) [ 0.662241] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.665816] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.667480] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.669484] NET: Registered protocol family 1 [ 0.671698] RPC: Registered named UNIX socket transport module. [ 0.673816] RPC: Registered udp transport module. [ 0.675465] RPC: Registered tcp transport module. [ 0.677240] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.679030] NET: Registered protocol family 44 [ 0.680606] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.682618] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.684648] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.686148] PCI: CLS 0 bytes, default 64 [ 0.687354] Unpacking initramfs... [ 2.042103] debug: unmapping init [mem 0xffff8ac83cc64000-0xffff8ac83ffcffff] [ 2.046074] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.048355] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.051145] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.565487] Initialise system trusted keyrings [ 2.567375] Key type blacklist registered [ 2.569405] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.579122] zbud: loaded [ 2.582596] *** VALIDATE nfs *** [ 2.583910] *** VALIDATE nfs4 *** [ 2.585550] pstore: using deflate compression [ 2.589508] Platform Keyring initialized [ 2.691425] NET: Registered protocol family 38 [ 2.693431] Key type asymmetric registered [ 2.694836] Asymmetric key parser 'x509' registered [ 2.696264] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.698903] io scheduler mq-deadline registered [ 2.700855] io scheduler kyber registered [ 2.702517] io scheduler bfq registered [ 2.704399] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.706793] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.710214] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.714231] ACPI: Power Button [PWRF] [ 2.720325] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.727291] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.740809] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.768357] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.797049] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.800489] Non-volatile memory driver v1.3 [ 2.801652] Linux agpgart interface v0.103 [ 2.831395] virtio_blk virtio1: [vda] 135184 512-byte logical blocks (69.2 MB/66.0 MiB) [ 2.833263] vda: detected capacity change from 0 to 69214208 [ 2.847844] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.850468] vdb: detected capacity change from 0 to 1073741824 [ 2.857213] libphy: Fixed MDIO Bus: probed [ 2.896530] usbcore: registered new interface driver usbserial_generic [ 2.899772] usbserial: USB Serial support registered for generic [ 2.901777] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.906221] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.908169] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.910960] mousedev: PS/2 mouse device common for all mice [ 2.918588] rtc_cmos 00:05: RTC can wake from S4 [ 2.921117] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.923721] rtc_cmos 00:05: registered as rtc0 [ 2.929877] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.931113] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.933973] intel_pstate: CPU model not supported [ 2.940396] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.942374] hid: raw HID events driver (C) Jiri Kosina [ 2.946600] usbcore: registered new interface driver usbhid [ 2.948685] usbhid: USB HID core driver [ 2.950322] drop_monitor: Initializing network drop monitor service [ 2.953054] Initializing XFRM netlink socket [ 2.954814] NET: Registered protocol family 10 [ 2.957552] Segment Routing with IPv6 [ 2.959091] NET: Registered protocol family 17 [ 2.960825] mpls_gso: MPLS GSO support [ 2.967305] RAS: Correctable Errors collector initialized. [ 2.970058] AVX version of gcm_enc/dec engaged. [ 2.971073] AES CTR mode by8 optimization enabled [ 3.042627] sched_clock: Marking stable (3042595542, 0)->(3937493651, -894898109) [ 3.045309] registered taskstats version 1 [ 3.047305] Loading compiled-in X.509 certificates [ 3.049793] zswap: loaded using pool lzo/zbud [ 3.075939] Key type big_key registered [ 3.089812] Key type encrypted registered [ 3.091524] ima: No TPM chip found, activating TPM-bypass! [ 3.093864] ima: Allocated hash algorithm: sha1 [ 3.095370] ima: No architecture policies found [ 3.096672] evm: Initialising EVM extended attributes: [ 3.097916] evm: security.selinux [ 3.098812] evm: security.ima [ 3.099623] evm: security.capability [ 3.100504] evm: HMAC attrs: 0x1 [ 3.103624] rtc_cmos 00:05: setting system clock to 2026-03-30 16:08:37 UTC (1774886917) [ 3.108414] debug: unmapping init [mem 0xffffffff8d403000-0xffffffff8d5fffff] [ 3.110839] debug: unmapping init [mem 0xffffffff8c182000-0xffffffff8c458fff] [ 3.120083] Write protecting the kernel read-only data: 28672k [ 3.122920] debug: unmapping init [mem 0xffffffff8a803000-0xffffffff8a9fffff] [ 3.125163] debug: unmapping init [mem 0xffffffff8b114000-0xffffffff8b1fffff] [ 3.157798] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.164327] systemd[1]: Detected virtualization kvm. [ 3.165614] systemd[1]: Detected architecture x86-64. [ 3.167457] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.194617] systemd[1]: No hostname configured. [ 3.196033] systemd[1]: Set hostname to . [ 3.197870] random: systemd: uninitialized urandom read (16 bytes read) [ 3.199650] systemd[1]: Initializing machine ID from random generator. [ 3.253165] random: ln: uninitialized urandom read (6 bytes read) [ 3.358387] random: systemd: uninitialized urandom read (16 bytes read) [ 3.360678] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.364159] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.368411] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.054452] device-mapper: uevent: version 1.0.3 [ 4.056245] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.825502] virtio_net virtio0 ens2: renamed from eth0 [ 4.902890] scsi host0: ata_piix [ 4.930270] scsi host1: ata_piix [ 4.932145] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.934604] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.505867] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.606023] random: crng init done [ 9.608750] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.130349] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.315983] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.502316] SELinux: Disabled at runtime. [ 12.635704] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.649828] systemd[1]: Detected virtualization kvm. [ 12.653887] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.284225] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.289471] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.313310] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.324904] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.334027] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.364139] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.388552] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ 15.742706] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 16.707503] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.881115] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.126275] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 18.934816] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 19.058634] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 23.399458] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 24.362568] NFS: Registering the id_resolver key type [ 24.365265] Key type id_resolver registered [ 24.368338] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ 33.393775] hrtimer: interrupt took 9254014 ns [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg206-client login: [ 108.281657] libcfs: loading out-of-tree module taints kernel. [ 108.620862] Key type ._llcrypt registered [ 108.639898] Key type .llcrypt registered [ 109.431701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 109.454874] alg: No test for adler32 (adler32-zlib) [ 111.029678] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 112.624672] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 114.575334] Key type lgssc registered [ 117.335859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 258.502810] Lustre: Mounted lustre-client [ 264.487814] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.130573] Lustre: lustre-OST0000-osc-ffff8ac887370000: disconnect after 23s idle [ 287.993932] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing check_logdir /tmp/testlogs/ [ 294.543120] Lustre: DEBUG MARKER: oleg206-client.virtnet: executing yml_node [ 299.662951] Lustre: DEBUG MARKER: Client: 2.17.50.165 [ 303.264700] Lustre: DEBUG MARKER: MDS: 2.17.50.165 [ 306.112205] Lustre: DEBUG MARKER: OSS: 2.17.50.165 [ 308.399570] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Mar 30 12:13:40 EDT 2026 [ 327.364423] LustreError: 7917:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8ac887370000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 327.395230] Lustre: Unmounted lustre-client [ 353.859943] Key type lgssc unregistered [ 354.166616] LNet: 8364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.176383] LNetError: 8364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 354.193054] LNet: Removed LNI 192.168.202.6@tcp [ 355.138235] Key type .llcrypt unregistered [ 355.144505] Key type ._llcrypt unregistered [ 382.622598] Key type ._llcrypt registered [ 382.625678] Key type .llcrypt registered [ 382.858935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 382.900295] alg: No test for adler32 (adler32-zlib) [ 383.953822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 384.052091] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 384.070706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 389.031679] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 389.108215] LNet: 9239:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.118466] LNetError: 9239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 389.133469] LNet: Removed LNI 192.168.202.6@tcp [ 389.843762] Key type .llcrypt unregistered [ 389.847085] Key type ._llcrypt unregistered [ 397.480585] Key type ._llcrypt registered [ 397.483251] Key type .llcrypt registered [ 398.991167] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:15:11 (1774887311) [ 399.510756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 399.549878] alg: No test for adler32 (adler32-zlib) [ 400.542609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.640939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.665345] LNet: 9658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.693408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.725325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 400.738346] LNet: 9665:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 408.898175] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:15:21 (1774887321) [ 409.015352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.048520] LNet: 10238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.086300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.187951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 416.878421] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:15:29 (1774887329) [ 417.046889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.076853] LNet: 10824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.103817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.187819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 424.512679] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:15:37 (1774887337) [ 424.590105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.616188] LNet: 11407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.641781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.731674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 432.353919] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:15:45 (1774887345) [ 432.535489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.588275] LNet: 11993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.629321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.722722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 433.460202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 441.628715] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:15:54 (1774887354) [ 441.833319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.861522] LNet: 12585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.900435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.090564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 442.779295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 443.730215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 451.627883] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:16:04 (1774887364) [ 451.977982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.019889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.100412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 459.646164] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:16:12 (1774887372) [ 459.798142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.830206] LNet: 13768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.843549] LNet: 13768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 459.861952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.966299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 460.011471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 460.667738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 460.724655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 461.395479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 461.480501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 462.118836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 462.205953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 462.777492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 462.840720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 463.457228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 463.504235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 464.073170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 464.114602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 464.605368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 464.658082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 464.996519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 465.060472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 465.617292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 465.651346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 466.000852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 466.038452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 466.486303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 466.527758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 466.989972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 467.056519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 474.158577] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:16:27 (1774887387) [ 474.509797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.581270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.680302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 474.723477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 481.512844] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:16:34 (1774887394) [ 481.712544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.741090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.830696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 481.858954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 488.999724] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:16:42 (1774887402) [ 489.190858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.287602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.357806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 489.403766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 496.822969] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:16:49 (1774887409) [ 497.296823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.316407] LNet: 16215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.322088] LNet: 16215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 497.348733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.441460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 497.529826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 505.002705] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:16:58 (1774887418) [ 505.264849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.316457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.448312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 505.488892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 512.813460] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:17:05 (1774887425) [ 512.900260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.969882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.152380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 513.235614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 520.509415] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:17:13 (1774887433) [ 520.855847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.913106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.053914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 521.552978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 529.034922] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:17:21 (1774887441) [ 529.386678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.470693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.770987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 530.074982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 538.216168] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:17:31 (1774887451) [ 538.521266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.570816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.693953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.174664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 547.546700] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:17:40 (1774887460) [ 548.059327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.110869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.643485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 548.900769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 557.519476] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:17:50 (1774887470) [ 557.653094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.711953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.904992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 557.983527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 567.983074] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:17:59 (1774887479) [ 568.202475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.289773] LNet: 21045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.304339] LNet: 21045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 568.351812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.554552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.676340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 580.911844] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:18:13 (1774887493) [ 581.185490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.240718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.304163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 591.123231] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:18:23 (1774887503) [ 591.450029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.499456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.607995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 600.006735] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:18:32 (1774887512) [ 600.126224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.237640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.386433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 601.136223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 611.620825] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:18:44 (1774887524) [ 612.007983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.082932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.210718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 612.292696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 619.829910] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:18:52 (1774887532) [ 620.090637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.187158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.286147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 620.370637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 628.307540] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:19:00 (1774887540) [ 628.427556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.528599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.670480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 628.706838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 636.632086] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:19:09 (1774887549) [ 636.957727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.008204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.083630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 637.115602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 643.880519] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:19:17 (1774887557) [ 644.385787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.440523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.502708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 644.614799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 644.637321] LNetError: 25779:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 644.697752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 644.751557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 651.570663] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:19:24 (1774887564) [ 651.915208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.001910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.109139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.UCvH4FQY7m [ 652.255044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.UCvH4FQY7m [ 652.341726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.UCvH4FQY7m [ 659.625608] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:19:32 (1774887572) [ 659.866549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.920931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.977180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 660.019372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 660.073530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 666.663159] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:19:39 (1774887579) [ 667.205209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.290950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.390743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 670.187464] LNet: Message forwarding will be enabled in 2s [ 673.247174] LNet: Message forwarding enabled [ 673.749213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 673.791838] LNet: Message forwarding will stop in 2s to 6s [ 678.544103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 678.830353] LNet: Message forwarding disabled [ 680.648387] LNet: Message forwarding will be enabled in 2s [ 683.935200] LNet: Message forwarding enabled [ 689.260281] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:20:01 (1774887601) [ 689.412577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.448737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.556951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 698.551853] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:20:11 (1774887611) [ 698.732706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.809385] LNet: 28804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.815779] LNet: 28804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 698.878054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.999809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 699.033773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 699.068658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 699.095301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 699.128230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 699.201699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 699.264621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 699.317276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 699.373095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 699.441558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 699.505860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 699.539628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 699.593351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 699.654110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 699.712603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 699.798539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 699.846861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 699.919413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 699.994768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 700.030748] LNetError: 28852:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 709.176327] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:20:21 (1774887621) [ 709.328646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 709.457172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.617693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 717.794448] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:20:30 (1774887630) [ 720.925292] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg206-server [ 723.902550] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:20:36 (1774887636) [ 724.361315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 724.479117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 724.585747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 724.624098] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 727.356168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[7-11]@tcp [ 727.407393] LNet: 30572:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 727.486418] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.8@tcp is being used as a gateway but routing feature is not turned on [ 727.518578] LNetError: 30361:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.8@tcp has gone from up to down [ 728.488947] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.7@tcp is being used as a gateway but routing feature is not turned on [ 728.513046] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 729.525135] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.9@tcp is being used as a gateway but routing feature is not turned on [ 729.544943] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 731.620695] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.7@tcp is being used as a gateway but routing feature is not turned on [ 731.656902] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 735.726199] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.7@tcp is being used as a gateway but routing feature is not turned on [ 735.742764] LNetError: 30361:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 736.973130] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:20:49 (1774887649) [ 738.790128] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg206-server [ 740.484827] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:20:53 (1774887653) [ 740.737455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.774511] LNetError: 31507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.793199] LNet: Removed LNI 192.168.202.6@tcp [ 740.836815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.875294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.903170] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 741.489794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[7-12/2]@tcp [ 741.522799] LNet: 31624:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 741.534218] LNet: 31624:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 741.561317] LNetError: 31511:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.7@tcp has gone from up to down [ 741.579860] LNetError: 31511:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 741.594173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[7-12/2]@tcp [ 749.084845] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:21:01 (1774887661) [ 749.446556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.487302] LNetError: 32218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.503607] LNet: Removed LNI 192.168.202.6@tcp [ 749.551593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.683331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 749.757317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.805811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.883782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 749.949753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 749.992714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.057541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 750.181198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.232731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.339365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 750.434304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.491832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.553689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 750.658581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.706693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.812496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 750.954370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.997321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.097464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 751.204669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.250951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.348686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 751.468769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.570339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.688373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 751.853049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.911252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.002729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 752.113655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.168694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.243957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 752.350125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.451060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.630608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 760.807030] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:21:13 (1774887673) [ 763.294705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg206-server [ 765.222683] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:21:18 (1774887678) [ 767.971738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg206-server [ 770.303779] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:21:23 (1774887683) [ 771.754642] Key type .llcrypt unregistered [ 771.758414] Key type ._llcrypt unregistered [ 779.189729] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.384312] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.402958] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.413614] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 782.642939] Key type ._llcrypt registered [ 782.648368] Key type .llcrypt registered [ 783.005627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 783.037808] alg: No test for adler32 (adler32-zlib) [ 784.168260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.309400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.356495] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 785.385487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 785.403285] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 788.645830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 788.681920] LNet: Removed LNI 10.1.2.3@tcp [ 788.867636] LNet: 34343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.874879] LNetError: 34343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.893204] LNet: Removed LNI 192.168.202.6@tcp [ 789.968186] Key type .llcrypt unregistered [ 789.972334] Key type ._llcrypt unregistered [ 806.865769] Key type ._llcrypt registered [ 806.867515] Key type .llcrypt registered [ 808.467885] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:22:01 (1774887721) [ 809.426869] Key type .llcrypt unregistered [ 809.431684] Key type ._llcrypt unregistered [ 817.468319] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 817.696515] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 817.708358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 817.737429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 820.475962] Key type ._llcrypt registered [ 820.483980] Key type .llcrypt registered [ 820.779982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 820.819334] alg: No test for adler32 (adler32-zlib) [ 821.732448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.797898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 821.841045] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 822.712666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 822.733358] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 825.843473] LNet: 35836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.851820] LNetError: 35836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.860083] LNet: Removed LNI 192.168.202.6@tcp [ 825.869994] LNet: Removed LNI 10.1.2.3@tcp [ 826.442581] Key type .llcrypt unregistered [ 826.444168] Key type ._llcrypt unregistered [ 841.968141] Key type ._llcrypt registered [ 841.979813] Key type .llcrypt registered [ 843.555135] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:22:36 (1774887756) [ 844.355463] Key type .llcrypt unregistered [ 844.357460] Key type ._llcrypt unregistered [ 851.787247] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 852.054868] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 852.062537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 852.066369] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 854.752745] Key type ._llcrypt registered [ 854.763766] Key type .llcrypt registered [ 854.920336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 854.937433] alg: No test for adler32 (adler32-zlib) [ 855.917074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.002960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 856.039717] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 857.766179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 857.782865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 859.564057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 859.588478] LNet: Removed LNI 10.1.2.3@tcp [ 859.611531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 859.640649] LNet: Removed LNI 10.1.2.31@tcp [ 859.657708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 859.677322] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 860.260462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 860.275641] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 861.517617] LNet: 37758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.528068] LNetError: 37758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 861.543615] LNet: Removed LNI 10.1.2.3@tcp [ 861.570598] LNet: Removed LNI 10.1.2.31@tcp [ 862.191661] Key type .llcrypt unregistered [ 862.193775] Key type ._llcrypt unregistered [ 876.319484] Key type ._llcrypt registered [ 876.321369] Key type .llcrypt registered [ 877.837852] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:23:10 (1774887790) [ 878.444978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 878.462328] alg: No test for adler32 (adler32-zlib) [ 879.471942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.547279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.593651] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 880.614244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.680661] LNet: 38713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.693854] LNetError: 38713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 880.745767] LNet: Removed LNI 192.168.202.6@tcp [ 880.794794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.943572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.6@tcp [ 880.994381] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 882.109397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.6@tcp [ 882.143496] LNet: Removed LNI 192.168.202.6@tcp [ 882.236554] LNet: 38848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.011534] Key type .llcrypt unregistered [ 883.013941] Key type ._llcrypt unregistered [ 898.023410] Key type ._llcrypt registered [ 898.027119] Key type .llcrypt registered [ 899.494203] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:23:32 (1774887812) [ 901.703893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg206-server [ 903.659879] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:23:36 (1774887816) [ 904.283673] Key type .llcrypt unregistered [ 904.286338] Key type ._llcrypt unregistered [ 910.281942] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.413607] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.431094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.438039] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 913.342534] Key type ._llcrypt registered [ 913.345697] Key type .llcrypt registered [ 913.609803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.651980] alg: No test for adler32 (adler32-zlib) [ 914.746832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.842430] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 915.793248] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 918.938714] LNet: 40688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.947989] LNetError: 40688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.956559] LNet: Removed LNI 192.168.202.6@tcp1 [ 918.973815] LNet: Removed LNI 10.1.2.3@tcp2 [ 919.745594] Key type .llcrypt unregistered [ 919.748047] Key type ._llcrypt unregistered [ 934.201294] Key type ._llcrypt registered [ 934.202729] Key type .llcrypt registered [ 935.380343] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:24:08 (1774887848) [ 935.863617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.883861] alg: No test for adler32 (adler32-zlib) [ 936.918145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.055633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 937.116388] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 938.462325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.487451] LNet: 41657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.497123] LNetError: 41657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.508930] LNet: Removed LNI 192.168.202.6@tcp [ 938.530756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.593303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 938.616286] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 946.385243] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:24:18 (1774887858) [ 946.694997] LNet: 42247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.721704] LNetError: 42247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.770160] LNet: Removed LNI 192.168.202.6@tcp [ 947.432417] Key type .llcrypt unregistered [ 947.434084] Key type ._llcrypt unregistered [ 954.709106] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 954.886013] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 954.892383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 954.902438] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 957.629697] Key type ._llcrypt registered [ 957.632316] Key type .llcrypt registered [ 957.811572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 957.840325] alg: No test for adler32 (adler32-zlib) [ 958.798575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.855397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 958.876179] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 959.620886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 959.644576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 961.762379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.782931] LNet: 42992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.789555] LNetError: 42992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.798559] LNet: Removed LNI 192.168.202.6@tcp [ 961.806778] LNet: Removed LNI 10.1.2.3@tcp [ 961.826185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.879745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 961.910803] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 961.926696] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 962.626224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.660871] LNet: 43019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.669993] LNetError: 43019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.676919] LNetError: 43019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 962.682374] LNet: Removed LNI 192.168.202.6@tcp [ 962.704941] LNet: Removed LNI 10.1.2.3@tcp [ 962.740386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.854488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 962.898066] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 962.910924] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 963.561642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 963.585582] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 965.020622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.036744] LNet: 43182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.043455] LNetError: 43182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.051581] LNetError: 43182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 965.056423] LNet: Removed LNI 192.168.202.6@tcp [ 965.063636] LNet: Removed LNI 10.1.2.3@tcp [ 965.082140] LNet: Removed LNI 192.168.202.6@tcp2 [ 965.121757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.200984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 965.225153] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 965.229620] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.236658] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 972.947704] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:24:46 (1774887886) [ 973.118473] LNet: 43779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.130502] LNetError: 43779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.141223] LNetError: 43779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 973.154233] LNet: Removed LNI 192.168.202.6@tcp [ 973.158949] LNet: Removed LNI 10.1.2.3@tcp [ 973.171249] LNet: Removed LNI 192.168.202.6@tcp2 [ 974.088863] Key type .llcrypt unregistered [ 974.091117] Key type ._llcrypt unregistered [ 980.508134] Key type ._llcrypt registered [ 980.510636] Key type .llcrypt registered [ 980.882937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 980.896494] alg: No test for adler32 (adler32-zlib) [ 981.912565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.013104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 982.067973] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 983.425958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.451776] LNet: 44175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.457607] LNetError: 44175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.469404] LNet: Removed LNI 192.168.202.6@tcp [ 983.497399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.561613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 983.582456] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 984.261927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.285501] LNet: 44205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.291817] LNetError: 44205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.304850] LNet: Removed LNI 192.168.202.6@tcp [ 984.326476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.379097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 984.428680] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 984.792679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.831764] LNet: Removed LNI 192.168.202.6@tcp [ 984.862138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.984957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 985.038693] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 985.675756] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 985.830139] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 985.841541] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 985.855344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 988.469440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.502622] LNet: 44313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.513068] LNet: 44313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 988.523620] LNetError: 44313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.534049] LNetError: 44313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 988.582546] LNet: Removed LNI 192.168.202.6@tcp [ 988.610481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.718476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 988.778305] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 996.214958] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:25:09 (1774887909) [ 996.502483] LNet: 44911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 996.507946] LNetError: 44911:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 996.524850] LNet: Removed LNI 192.168.202.6@tcp [ 997.130626] Key type .llcrypt unregistered [ 997.133547] Key type ._llcrypt unregistered [ 1004.792766] Key type ._llcrypt registered [ 1004.794872] Key type .llcrypt registered [ 1005.352928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1005.369956] alg: No test for adler32 (adler32-zlib) [ 1006.336709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.435788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1006.456426] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1007.298810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.318478] LNet: 45309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.324055] LNetError: 45309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.335924] LNet: Removed LNI 192.168.202.6@tcp [ 1007.354833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.419536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1007.458313] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1015.368972] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:25:28 (1774887928) [ 1015.715290] LNet: 45902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.721519] LNetError: 45902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.737718] LNet: Removed LNI 192.168.202.6@tcp [ 1016.578697] Key type .llcrypt unregistered [ 1016.584684] Key type ._llcrypt unregistered [ 1023.498693] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1023.727201] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1023.738738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1023.745172] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1026.483771] Key type ._llcrypt registered [ 1026.488492] Key type .llcrypt registered [ 1026.692991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1026.704046] alg: No test for adler32 (adler32-zlib) [ 1027.678981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.811379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1027.875311] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1028.923324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1028.941360] LNet: 46354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.948894] LNetError: 46354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.957717] LNet: Removed LNI 192.168.202.6@tcp [ 1028.976513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.024325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1029.052381] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1036.960629] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:25:49 (1774887949) [ 1037.352839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.393994] LNet: 46953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.401605] LNetError: 46953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.435951] LNet: Removed LNI 192.168.202.6@tcp [ 1037.455928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.564280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1046.021168] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:25:58 (1774887958) [ 1046.332539] LNet: 47536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.014185] Key type .llcrypt unregistered [ 1047.017403] Key type ._llcrypt unregistered [ 1054.016450] Key type ._llcrypt registered [ 1054.018864] Key type .llcrypt registered [ 1054.348469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.378710] alg: No test for adler32 (adler32-zlib) [ 1055.618456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.785810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1055.823984] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1057.273339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1057.314391] LNet: 47934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.329095] LNetError: 47934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.366829] LNet: Removed LNI 192.168.202.6@tcp [ 1057.391343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.503328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1057.572614] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1067.107306] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:26:20 (1774887980) [ 1067.253481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.299348] LNet: 48522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.320706] LNetError: 48522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.372929] LNet: Removed LNI 192.168.202.6@tcp [ 1067.435928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1067.583279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1076.095839] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:26:29 (1774887989) [ 1078.214947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1080.689461] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:26:33 (1774887993) [ 1083.271161] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1086.132220] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:26:38 (1774887998) [ 1088.096484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1090.119054] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:26:42 (1774888002) [ 1092.763369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1094.878440] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:26:47 (1774888007) [ 1096.637690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1098.667320] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:26:51 (1774888011) [ 1099.155844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1099.177373] LNet: 50583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.200934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1099.293284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1099.325309] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1100.105272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.6 [ 1100.131291] LNet: 50716:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1100.140098] LNet: Removed LNI 192.168.202.6@tcp [ 1106.556613] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:26:59 (1774888019) [ 1106.833687] LNet: 51287:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.852777] LNetError: 51287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.873801] LNet: Removed LNI 192.168.202.6@tcp [ 1107.559975] Key type .llcrypt unregistered [ 1107.563117] Key type ._llcrypt unregistered [ 1114.881213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1115.144503] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1115.154450] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1115.163312] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1118.077174] Key type ._llcrypt registered [ 1118.081510] Key type .llcrypt registered [ 1118.306178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1118.333529] alg: No test for adler32 (adler32-zlib) [ 1119.961915] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1127.648948] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:27:20 (1774888040) [ 1127.760931] LNet: 52185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.776931] LNetError: 52185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1127.827311] LNet: Removed LNI 10.1.2.3@tcp [ 1128.656334] Key type .llcrypt unregistered [ 1128.658337] Key type ._llcrypt unregistered [ 1137.077717] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1137.349781] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1137.357860] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1137.362641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1139.116731] Key type ._llcrypt registered [ 1139.120833] Key type .llcrypt registered [ 1139.418221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.435184] alg: No test for adler32 (adler32-zlib) [ 1140.513725] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1148.072159] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:27:40 (1774888060) [ 1148.593688] LNet: 53080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.606843] LNetError: 53080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1148.624447] LNet: Removed LNI 10.1.2.3@tcp [ 1149.406316] Key type .llcrypt unregistered [ 1149.407804] Key type ._llcrypt unregistered [ 1156.569563] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1156.910042] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1156.917906] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1156.922928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1158.735375] Key type ._llcrypt registered [ 1158.737710] Key type .llcrypt registered [ 1158.943438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1158.966949] alg: No test for adler32 (adler32-zlib) [ 1160.024573] LNet: 53374:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1160.040890] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1167.691414] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:28:00 (1774888080) [ 1167.866138] LNet: 53961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.874271] LNetError: 53961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1167.901043] LNet: Removed LNI 10.1.2.3@tcp [ 1168.652913] Key type .llcrypt unregistered [ 1168.656946] Key type ._llcrypt unregistered [ 1175.518229] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1175.815520] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1175.826344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1175.831175] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1178.763321] Key type ._llcrypt registered [ 1178.770206] Key type .llcrypt registered [ 1179.044240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1179.065746] alg: No test for adler32 (adler32-zlib) [ 1180.020255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1180.088397] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1189.765847] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:28:22 (1774888102) [ 1190.442964] LNet: 55197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.447944] LNetError: 55197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1190.460471] LNet: Removed LNI 10.1.2.3@tcp [ 1191.163308] Key type .llcrypt unregistered [ 1191.164969] Key type ._llcrypt unregistered [ 1199.101187] Key type ._llcrypt registered [ 1199.104356] Key type .llcrypt registered [ 1199.365733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1199.389944] alg: No test for adler32 (adler32-zlib) [ 1200.415991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1200.446752] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1210.675294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 1211.674548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_interrupt -r 1 [ 1211.725391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_interrupt -r 1 [ 1211.773538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1211.805787] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1212.104498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.136202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1212.173363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.200876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.230641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.258678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.284418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.311551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.333624] LNet: 55636:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.357520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.384243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.415902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.446333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.474114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.505840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.538043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1213.010782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_dropped -r 1 [ 1213.042861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_dropped -r 1 [ 1213.079229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1213.096285] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1213.358363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1213.406984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1213.450574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.478114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.518377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.562317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.601966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.656224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.677818] LNet: 55745:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.716777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.779386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.793172] LNet: 1 local NIs in recovery (showing 1): 192.168.202.6@tcp [ 1213.832424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.874522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.921563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.956366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.003948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.692192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_aborted -r 1 [ 1214.736548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_aborted -r 1 [ 1214.769604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1214.783238] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1214.996428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1215.018783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1215.044659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.073594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.095628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1215.121389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1215.158349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1215.187362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1215.219360] LNet: 55854:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1215.250662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1215.281374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.318738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.355816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.377760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.407754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.434531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1215.908646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_no_route -r 1 [ 1215.945147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_no_route -r 1 [ 1215.978175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1216.232822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.276826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1216.309257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.342550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.404276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.442768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.487833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.520668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.536168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.560538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.590623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.616535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1216.659939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.684739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.712459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1217.319473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_timeout -r 1 [ 1217.363464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_timeout -r 1 [ 1217.438549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1217.472778] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1217.475681] LNet: Skipped 2 previous similar messages [ 1217.720549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1217.764203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1217.810560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1217.849590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1217.904553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1217.958496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1218.010941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1218.040123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1218.055525] LNet: 56073:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1218.064309] LNet: 56073:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 1218.076159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1218.106312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1218.131290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.158570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1218.189978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1218.215345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1218.258855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1218.746903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_error -r 1 [ 1218.786932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_error -r 1 [ 1218.814730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1219.019625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1219.052790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1219.080783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1219.109876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1219.149817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1219.185619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1219.252951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1219.287624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1219.323749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1219.358633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1219.392773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1219.419531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.443326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.489453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.554523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1219.911173] LNet: 56223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.923380] LNetError: 56223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.950369] LNet: Removed LNI 192.168.202.6@tcp [ 1220.499794] Key type .llcrypt unregistered [ 1220.502810] Key type ._llcrypt unregistered [ 1239.673398] Key type ._llcrypt registered [ 1239.677317] Key type .llcrypt registered [ 1241.404630] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:29:13 (1774888153) [ 1242.411440] Key type .llcrypt unregistered [ 1242.415798] Key type ._llcrypt unregistered [ 1249.176411] Key type ._llcrypt registered [ 1249.181364] Key type .llcrypt registered [ 1249.325331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1249.337430] alg: No test for adler32 (adler32-zlib) [ 1250.273635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1250.296517] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1258.784734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 1272.701543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.743226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1272.787576] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1274.213782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_interrupt -r 1 [ 1274.254805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_interrupt -r 1 [ 1274.283558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_interrupt -r 1 [ 1274.332885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_interrupt -r 1 [ 1274.385268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_interrupt -r 1 [ 1274.435671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_interrupt -r 1 [ 1274.493567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_interrupt -r 1 [ 1274.565572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_interrupt -r 1 [ 1274.615146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1274.630358] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1274.635355] LNet: 1 local NIs in recovery (showing 1): 192.168.202.6@tcp [ 1274.837173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1274.864124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1274.889379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1274.918825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1274.948615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1274.972519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.000313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1275.023774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1275.036314] LNet: 57693:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.049599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.078841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.113365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.153700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1275.193705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1275.241996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1275.284824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1275.893109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_dropped -r 1 [ 1275.939587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_dropped -r 1 [ 1275.990780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_dropped -r 1 [ 1276.036952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_dropped -r 1 [ 1276.106632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_dropped -r 1 [ 1276.172190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_dropped -r 1 [ 1276.235230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_dropped -r 1 [ 1276.333746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_dropped -r 1 [ 1276.390655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1276.416218] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1276.424707] LNet: Skipped 2 previous similar messages [ 1276.732535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1276.776476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1276.820736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.863717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1276.918953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.960818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.007322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.051112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1277.088550] LNet: 57815:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1277.115085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1277.164125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1277.210213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1277.243085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1277.282651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1277.307763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1277.339934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1277.867631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_aborted -r 1 [ 1277.895598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_aborted -r 1 [ 1277.926895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_aborted -r 1 [ 1277.959585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_aborted -r 1 [ 1277.997419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_aborted -r 1 [ 1278.028559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_aborted -r 1 [ 1278.081306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_aborted -r 1 [ 1278.139493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_aborted -r 1 [ 1278.200371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1278.238792] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1278.250870] LNet: Skipped 2 previous similar messages [ 1278.590877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.623652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1278.658865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.703947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.748325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.801943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.837712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.882984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.924361] LNet: 57936:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.950654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1279.004343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1279.061235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.109435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.160800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.212702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1279.264440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.928736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_no_route -r 1 [ 1279.966542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_no_route -r 1 [ 1280.012546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_no_route -r 1 [ 1280.057544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_no_route -r 1 [ 1280.098672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_no_route -r 1 [ 1280.143869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_no_route -r 1 [ 1280.186343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_no_route -r 1 [ 1280.236427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_no_route -r 1 [ 1280.284861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1280.309244] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1280.312540] LNet: Skipped 2 previous similar messages [ 1280.616718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.649698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1280.698984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.760838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.807557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.855148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.885837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.917247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.939261] LNet: 58057:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.949983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.974165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.998213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1281.036310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1281.066910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1281.097884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1281.121627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1281.531963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_timeout -r 1 [ 1281.557149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_timeout -r 1 [ 1281.598950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_timeout -r 1 [ 1281.645031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_timeout -r 1 [ 1281.683920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_timeout -r 1 [ 1281.739988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_timeout -r 1 [ 1281.804251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_timeout -r 1 [ 1281.866419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_timeout -r 1 [ 1281.925635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1282.382266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1282.441610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1282.526357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1282.608447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1282.689603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1282.764933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1282.823508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1282.869852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1282.898253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1282.980145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.044915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.092312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.141707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.211397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.266962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1283.981356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_error -r 1 [ 1284.014699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_error -r 1 [ 1284.053622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_error -r 1 [ 1284.116364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_error -r 1 [ 1284.195655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_error -r 1 [ 1284.270568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_error -r 1 [ 1284.335279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_error -r 1 [ 1284.404183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_error -r 1 [ 1284.474750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1284.530248] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1284.559170] LNet: Skipped 7 previous similar messages [ 1284.873489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1284.924612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1284.960899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1285.005630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1285.040836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1285.101682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1285.134744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1285.161752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1285.180213] LNet: 58300:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1285.197118] LNet: 58300:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 1285.228585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1285.276221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1285.364858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1285.417485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1285.458541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1285.517086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1285.575203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1292.149576] LNet: 58376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.157687] LNetError: 58376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1292.167291] LNet: Removed LNI 192.168.202.6@tcp [ 1293.222252] LNet: Removed LNI 192.168.202.6@tcp1 [ 1293.690213] Key type .llcrypt unregistered [ 1293.691898] Key type ._llcrypt unregistered [ 1314.566797] Key type ._llcrypt registered [ 1314.571925] Key type .llcrypt registered [ 1315.855518] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:30:28 (1774888228) [ 1317.063627] Key type .llcrypt unregistered [ 1317.065797] Key type ._llcrypt unregistered [ 1324.706178] Key type ._llcrypt registered [ 1324.711747] Key type .llcrypt registered [ 1325.058609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1325.074836] alg: No test for adler32 (adler32-zlib) [ 1326.137251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1326.190151] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1334.641779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 1335.342611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e remote_dropped -r 1 [ 1335.369178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e remote_dropped -r 1 [ 1335.407577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1335.422159] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1335.685954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1335.765721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1335.815441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1335.863637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1335.889737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1335.923555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1335.978806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1336.020521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1336.050384] LNet: 59627:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1336.091719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1336.124798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1336.167792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1336.231534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1336.279571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.332162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1336.354556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1336.839406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e remote_error -r 1 [ 1336.867479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e remote_error -r 1 [ 1336.892544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1336.910387] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1337.194546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1337.230414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1337.276797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1337.306886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1337.340379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1337.361726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1337.396333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1337.442949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1337.467687] LNet: 59736:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1337.497790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1337.551211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1337.604992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1337.654621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1337.711534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1337.765167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1337.824235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1338.396395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e remote_timeout -r 1 [ 1338.430237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e remote_timeout -r 1 [ 1338.456726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1338.477452] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1338.764196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.796081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1338.855442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.918614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.960714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1339.030708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1339.107312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1339.140449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1339.164325] LNet: 59846:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1339.181221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1339.233857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.282878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.352152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.389865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.467989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.517574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1339.959913] LNet: 59887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1339.967742] LNetError: 59887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1339.998574] LNet: Removed LNI 192.168.202.6@tcp [ 1340.490446] Key type .llcrypt unregistered [ 1340.494172] Key type ._llcrypt unregistered [ 1361.812793] Key type ._llcrypt registered [ 1361.817408] Key type .llcrypt registered [ 1363.653529] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:31:15 (1774888275) [ 1364.958831] Key type .llcrypt unregistered [ 1364.966994] Key type ._llcrypt unregistered [ 1373.167596] Key type ._llcrypt registered [ 1373.168853] Key type .llcrypt registered [ 1373.472496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1373.506382] alg: No test for adler32 (adler32-zlib) [ 1374.539139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1374.600752] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1384.708577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 1398.282866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1398.358329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1398.390824] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1400.650547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e remote_dropped -r 1 [ 1400.680918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e remote_dropped -r 1 [ 1400.734134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e remote_dropped -r 1 [ 1400.806683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e remote_dropped -r 1 [ 1400.881308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e remote_dropped -r 1 [ 1400.988969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e remote_dropped -r 1 [ 1401.087695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e remote_dropped -r 1 [ 1401.166395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e remote_dropped -r 1 [ 1401.205747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1401.264086] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1401.279419] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.106@tcp [ 1401.664567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1401.722931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1401.782645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1401.835908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1401.895410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1402.051199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1402.116878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1402.146700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1402.190425] LNet: 61359:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1402.218136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1402.248422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1402.285395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1402.317391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1402.336728] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1402.347974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1402.351498] LNet: Skipped 3 previous similar messages [ 1402.397805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1402.440873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1402.511491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1403.140830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e remote_error -r 1 [ 1403.180479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e remote_error -r 1 [ 1403.211969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e remote_error -r 1 [ 1403.278793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e remote_error -r 1 [ 1403.331724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e remote_error -r 1 [ 1403.379719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e remote_error -r 1 [ 1403.419211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e remote_error -r 1 [ 1403.456583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e remote_error -r 1 [ 1403.485879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1403.499726] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1403.696533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1403.722487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1403.747788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1403.772648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1403.798483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1403.819665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1403.851495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1403.881718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1403.901452] LNet: 61482:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1403.924670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1403.965413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1404.008528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1404.052038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1404.074426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1404.101724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1404.134874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1404.161903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1404.680754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e remote_timeout -r 1 [ 1404.728784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e remote_timeout -r 1 [ 1404.758444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e remote_timeout -r 1 [ 1404.788105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e remote_timeout -r 1 [ 1404.826676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e remote_timeout -r 1 [ 1404.864766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e remote_timeout -r 1 [ 1404.926719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e remote_timeout -r 1 [ 1404.967699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e remote_timeout -r 1 [ 1405.036315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.106@tcp [ 1405.317310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1405.348147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1405.380822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1405.411851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1405.443782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1405.470820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1405.504390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1405.554054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1405.594662] LNet: 61606:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1405.618480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1405.655581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1405.688131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1405.731178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1405.765600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1405.805730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1405.856977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1405.884116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1411.393302] LNet: 61684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1411.398904] LNetError: 61684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.416410] LNet: Removed LNI 192.168.202.6@tcp [ 1411.426631] LNet: Removed LNI 192.168.202.6@tcp1 [ 1411.812851] Key type .llcrypt unregistered [ 1411.818250] Key type ._llcrypt unregistered [ 1429.749220] Key type ._llcrypt registered [ 1429.755847] Key type .llcrypt registered [ 1431.167467] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:32:23 (1774888343) [ 1431.713931] Key type .llcrypt unregistered [ 1431.715770] Key type ._llcrypt unregistered [ 1439.042056] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1439.228398] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1439.241191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1439.250392] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1441.898910] Key type ._llcrypt registered [ 1441.909546] Key type .llcrypt registered [ 1442.154847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1442.180738] alg: No test for adler32 (adler32-zlib) [ 1443.124112] LNet: 62766:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1443.134288] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1443.162993] LNet: 62780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.167923] LNetError: 62780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1443.183604] LNet: Removed LNI 192.168.202.6@tcp [ 1443.773519] Key type .llcrypt unregistered [ 1443.775056] Key type ._llcrypt unregistered [ 1450.551603] Key type ._llcrypt registered [ 1450.560522] Key type .llcrypt registered [ 1450.804713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1450.836942] alg: No test for adler32 (adler32-zlib) [ 1451.806448] LNet: 63039:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1451.816840] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1451.821648] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1451.869049] LNet: 63053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.882906] LNetError: 63053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.889557] LNet: Removed LNI 192.168.202.6@tcp [ 1451.909875] LNet: Removed LNI 10.1.2.3@tcp1 [ 1452.554422] Key type .llcrypt unregistered [ 1452.556058] Key type ._llcrypt unregistered [ 1459.329541] Key type ._llcrypt registered [ 1459.332907] Key type .llcrypt registered [ 1459.567342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1459.587218] alg: No test for adler32 (adler32-zlib) [ 1460.689264] LNet: 63313:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1460.713866] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1460.717167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1460.797443] LNet: 63327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.802223] LNetError: 63327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.810295] LNet: Removed LNI 192.168.202.6@tcp [ 1460.830410] LNet: Removed LNI 10.1.2.3@tcp [ 1461.784646] Key type .llcrypt unregistered [ 1461.791786] Key type ._llcrypt unregistered [ 1468.892774] Key type ._llcrypt registered [ 1468.895676] Key type .llcrypt registered [ 1469.331724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1469.343413] alg: No test for adler32 (adler32-zlib) [ 1470.282974] LNet: 63588:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1470.293645] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1470.296068] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1470.380103] LNet: 63602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.385965] LNetError: 63602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.393265] LNet: Removed LNI 192.168.202.6@tcp [ 1470.409323] LNet: Removed LNI 10.1.2.3@tcp [ 1471.193819] Key type .llcrypt unregistered [ 1471.196284] Key type ._llcrypt unregistered [ 1478.067712] Key type ._llcrypt registered [ 1478.069520] Key type .llcrypt registered [ 1478.316303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1478.348214] alg: No test for adler32 (adler32-zlib) [ 1479.288266] LNet: 63867:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1479.300702] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1479.303295] LNetError: 63867:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1479.308235] LNetError: Error -100 starting up LNI tcp [ 1479.318388] LNet: Removed LNI 192.168.202.6@tcp [ 1479.889415] Key type .llcrypt unregistered [ 1479.891112] Key type ._llcrypt unregistered [ 1490.936311] Key type ._llcrypt registered [ 1490.938469] Key type .llcrypt registered [ 1492.247477] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:33:25 (1774888405) [ 1493.246819] Key type .llcrypt unregistered [ 1493.250438] Key type ._llcrypt unregistered [ 1500.311581] Key type ._llcrypt registered [ 1500.314370] Key type .llcrypt registered [ 1500.761217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1500.815902] alg: No test for adler32 (adler32-zlib) [ 1501.782087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1501.829387] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1510.481511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 1511.404311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e network_timeout -r 1 [ 1511.452674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e network_timeout -r 1 [ 1511.511849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 1511.542833] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1511.824407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1511.854418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1511.883565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1511.928970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1511.989792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1512.019923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1512.061527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1512.095403] LNet: 1 local NIs in recovery (showing 1): 192.168.202.6@tcp [ 1512.101536] LNet: There was an unexpected network error while writing to 192.168.202.6: rc = -22 [ 1512.105725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1512.124363] LNet: 65124:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1512.150420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1512.234545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1512.282824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1512.350666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1512.422429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1512.514955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1512.927465] LNet: 65163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.944975] LNetError: 65163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1512.973947] LNet: Removed LNI 192.168.202.6@tcp [ 1513.661554] Key type .llcrypt unregistered [ 1513.666414] Key type ._llcrypt unregistered [ 1535.131177] Key type ._llcrypt registered [ 1535.133351] Key type .llcrypt registered [ 1535.328814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1535.385787] alg: No test for adler32 (adler32-zlib) [ 1536.338790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1536.363789] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1544.287663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 1555.889763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.018122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1556.071822] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1558.662292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e network_timeout -r 1 [ 1558.718913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e network_timeout -r 1 [ 1558.761789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e network_timeout -r 1 [ 1558.820213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e network_timeout -r 1 [ 1558.861215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e network_timeout -r 1 [ 1558.900945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e network_timeout -r 1 [ 1558.974342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e network_timeout -r 1 [ 1559.103318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e network_timeout -r 1 [ 1559.148466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 1559.188942] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1559.597133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1559.652438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1559.707912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1559.764473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1559.794666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1559.838440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1559.880726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1559.931734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1559.949362] LNet: 66046:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1559.967660] LNet: 1 local NIs in recovery (showing 1): 192.168.202.6@tcp [ 1559.970109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1559.971750] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.106@tcp [ 1560.004963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1560.058748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1560.139238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1560.195348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1560.254029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1565.742948] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1566.175209] LNet: 66120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.186183] LNetError: 66120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.193516] LNet: Removed LNI 192.168.202.6@tcp [ 1566.213877] LNet: Removed LNI 192.168.202.6@tcp1 [ 1566.585624] Key type .llcrypt unregistered [ 1566.588328] Key type ._llcrypt unregistered [ 1585.876585] Key type ._llcrypt registered [ 1585.879394] Key type .llcrypt registered [ 1587.123195] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:35:00 (1774888500) [ 1587.817619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1587.829668] alg: No test for adler32 (adler32-zlib) [ 1588.717220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.761233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.779450] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1589.385912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1589.403896] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1590.271336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 1590.341913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1590.405560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1590.434917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1590.474251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1590.509354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1590.537226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.6@tcp [ 1590.551626] LNetError: 67221:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.202.6@tcp to recovery [ 1590.819325] LNet: 1 local NIs in recovery (showing 1): 192.168.202.6@tcp [ 1590.820890] LNet: There was an unexpected network error while writing to 192.168.202.6: rc = -22 [ 1592.863331] LNet: There was an unexpected network error while writing to 192.168.202.6: rc = -22 [ 1593.974409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1596.964531] LNet: There was an unexpected network error while writing to 192.168.202.6: rc = -22 [ 1597.743464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1597.797868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1597.886865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.908318] LNet: 67328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.917695] LNetError: 67328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.925440] LNet: Removed LNI 192.168.202.6@tcp [ 1597.938059] LNet: Removed LNI 192.168.202.6@tcp1 [ 1597.964800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.016211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1598.062131] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1598.791791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1598.809683] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1599.492664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 1599.522825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1599.576957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1599.610888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1599.640678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1599.668474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1599.696914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1599.725454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.6@tcp [ 1599.740484] LNetError: 67600:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.202.6@tcp to recovery [ 1600.031538] LNet: There was an unexpected network error while writing to 192.168.202.6: rc = -22 [ 1603.232791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1606.176366] LNet: There was an unexpected network error while writing to 192.168.202.6: rc = -22 [ 1606.180488] LNet: Skipped 1 previous similar message [ 1611.387347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1611.407784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1611.430403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1618.371612] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:35:31 (1774888531) [ 1618.542466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.579727] LNet: 68314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.589673] LNetError: 68314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1618.598526] LNetError: 68314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1618.603797] LNet: Removed LNI 192.168.202.6@tcp [ 1618.617060] LNet: Removed LNI 192.168.202.6@tcp1 [ 1618.662671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.740469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.809168] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1619.673561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1619.690645] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1620.917731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 1621.055490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1621.110979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1621.166058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1621.197601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1621.232738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1621.253473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.6@tcp --health 0 [ 1621.346840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1621.791953] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.6@tcp [ 1622.815608] LNet: There was an unexpected network error while writing to 192.168.202.6: rc = -22 [ 1622.837465] LNet: Skipped 1 previous similar message [ 1637.134486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1637.281976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1637.305810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1637.329415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.6@tcp --health 500 [ 1637.400654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1644.083735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1644.202212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.241680] LNet: 68810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1644.259308] LNetError: 68810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1644.264973] LNetError: 68810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1644.274563] LNet: Removed LNI 192.168.202.6@tcp [ 1644.292731] LNet: Removed LNI 192.168.202.6@tcp1 [ 1644.336832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.440132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.480121] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1645.495343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1645.518062] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1646.582481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 1646.639072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1646.718693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1646.764964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1646.812923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1646.853684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1646.898588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1646.941840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.6@tcp --health 0 [ 1647.455805] LNet: There was an unexpected network error while writing to 192.168.202.6: rc = -22 [ 1647.459039] LNet: Skipped 3 previous similar messages [ 1648.494923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1659.022538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1659.069965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1659.114342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1668.940980] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:36:21 (1774888581) [ 1669.631571] LNet: 69855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.642962] LNetError: 69855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1669.652817] LNetError: 69855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1669.667588] LNet: Removed LNI 192.168.202.6@tcp [ 1669.685899] LNet: Removed LNI 192.168.202.6@tcp1 [ 1670.700028] Key type .llcrypt unregistered [ 1670.704704] Key type ._llcrypt unregistered [ 1677.070291] Key type ._llcrypt registered [ 1677.078488] Key type .llcrypt registered [ 1677.308812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1677.333254] alg: No test for adler32 (adler32-zlib) [ 1678.280492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1678.307153] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1687.188814] Lustre: DEBUG MARKER: Initial discovery [ 1687.509440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.106@tcp [ 1690.315143] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1690.382860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1690.422547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.106@tcp [ 1690.450146] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1690.472078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1690.531625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.106@tcp -r 1 -m GET -l 3 [ 1690.592141] LNet: 1 local NIs in recovery (showing 1): 192.168.202.6@tcp [ 1690.605815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.106@tcp -r 1 -m GET -e local_error [ 1690.653150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.106@tcp -r 1 -m PUT -l 6 [ 1692.068390] Lustre: DEBUG MARKER: Force oleg206-client.virtnet to discover 192.168.202.106@tcp (in background) [ 1692.375624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.106@tcp [ 1693.652378] Lustre: DEBUG MARKER: Force oleg206-server to discover 192.168.202.6@tcp [ 1695.455557] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1696.257825] Lustre: DEBUG MARKER: Wait for 70619 [ 1698.048170] Lustre: DEBUG MARKER: Finished wait on 70619 [ 1698.343248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1698.384388] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1698.400873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1698.793680] LNet: 71068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1698.815522] LNetError: 71068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.896514] LNet: Removed LNI 192.168.202.6@tcp [ 1700.448492] Key type .llcrypt unregistered [ 1700.450323] Key type ._llcrypt unregistered [ 1719.073092] Key type ._llcrypt registered [ 1719.079604] Key type .llcrypt registered [ 1720.563364] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:37:13 (1774888633) [ 1721.290353] Key type .llcrypt unregistered [ 1721.298234] Key type ._llcrypt unregistered [ 1727.657405] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1727.916047] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1727.923473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1727.936420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1730.676876] Key type ._llcrypt registered [ 1730.679362] Key type .llcrypt registered [ 1730.912281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1730.940873] alg: No test for adler32 (adler32-zlib) [ 1731.983534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.086145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.123628] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1732.778951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1732.797523] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1743.806747] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:37:36 (1774888656) [ 1743.945283] LNet: 73138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1743.962208] LNetError: 73138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1743.976834] LNet: Removed LNI 192.168.202.6@tcp [ 1744.005831] LNet: Removed LNI 10.1.2.3@tcp [ 1744.851332] Key type .llcrypt unregistered [ 1744.854301] Key type ._llcrypt unregistered [ 1752.758189] Key type ._llcrypt registered [ 1752.762329] Key type .llcrypt registered [ 1752.986537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1753.001181] alg: No test for adler32 (adler32-zlib) [ 1753.985360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.087843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.113332] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1754.972669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1755.011216] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1756.282385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.6@tcp [ 1757.343837] LNetError: Unexpected error -22 connecting to 192.168.202.6@tcp at host 192.168.202.6:988 [ 1763.405110] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:37:55 (1774888675) [ 1763.743068] LNet: 74281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1763.751616] LNetError: 74281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.760795] LNet: Removed LNI 192.168.202.6@tcp [ 1763.774549] LNet: Removed LNI 10.1.2.3@tcp [ 1764.428528] Key type .llcrypt unregistered [ 1764.432133] Key type ._llcrypt unregistered [ 1773.755120] Key type ._llcrypt registered [ 1773.758598] Key type .llcrypt registered [ 1774.069882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1774.108945] alg: No test for adler32 (adler32-zlib) [ 1775.208854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.310362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1775.356859] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1776.639617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1776.674979] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 1778.033405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.6@tcp1 --nid 192.168.202.6@tcp2 [ 1783.759702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.6@tcp1 192.168.202.6@tcp2 [ 1783.789811] LNetError: 75117:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.6@tcp1 to 192.168.202.6@tcp2 - Route table may be misconfigured [ 1783.836147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.6@tcp2 192.168.202.6@tcp1 [ 1792.855914] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:38:25 (1774888705) [ 1793.212968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.262054] LNet: 75690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.286942] LNetError: 75690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1793.299625] LNet: Removed LNI 192.168.202.6@tcp1 [ 1794.353215] LNet: Removed LNI 192.168.202.6@tcp2 [ 1794.409861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.492710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.538325] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1795.728238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1795.747790] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1796.639376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 1796.677994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.6@tcp 192.168.202.6@tcp [ 1796.713379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.6@tcp1 192.168.202.6@tcp1 [ 1796.758423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.6@tcp -d 192.168.202.6@tcp -e network_timeout [ 1796.791037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -e network_timeout [ 1796.823198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -e network_timeout [ 1796.859385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -e network_timeout [ 1796.900256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1796.923181] LNet: There was an unexpected network error while writing to 192.168.202.6: rc = -22 [ 1796.945213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1797.025231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1797.140583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1797.536950] LNet: 1 local NIs in recovery (showing 1): 192.168.202.6@tcp [ 1805.315433] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:38:37 (1774888717) [ 1805.466092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.506380] LNet: 76545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.513642] LNetError: 76545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.522829] LNetError: 76545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1805.543870] LNet: Removed LNI 192.168.202.6@tcp [ 1806.569120] LNet: Removed LNI 192.168.202.6@tcp1 [ 1806.628600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.767923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1807.323881] LNet: 76589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.030981] Key type .llcrypt unregistered [ 1808.034111] Key type ._llcrypt unregistered [ 1824.120380] Key type ._llcrypt registered [ 1824.126804] Key type .llcrypt registered [ 1825.734436] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:38:58 (1774888738) [ 1826.275691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.308965] alg: No test for adler32 (adler32-zlib) [ 1827.421281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.435768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1831.814492] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:39:04 (1774888744) [ 1831.968335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1832.003248] LNet: 77680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.042639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.179346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1832.262490] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1833.061862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1833.076234] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1833.941454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1833.968396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp1 [ 1834.001648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp1 [ 1841.568223] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:39:14 (1774888754) [ 1843.606905] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg206-server [ 1845.358612] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:39:18 (1774888758) [ 1847.596691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg206-server [ 1849.873808] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:39:22 (1774888762) [ 1852.290879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg206-server [ 1854.780546] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:39:27 (1774888767) [ 1856.628199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg206-server [ 1859.155517] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:39:31 (1774888771) [ 1861.882341] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg206-server [ 1864.785812] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:39:36 (1774888776) [ 1867.057350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg206-server [ 1870.477287] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:39:42 (1774888782) [ 1872.490294] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg206-server [ 1875.593857] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:39:47 (1774888787) [ 1878.914810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg206-server [ 1882.447121] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:39:53 (1774888793) [ 1883.906525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg206-server [ 1886.218771] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:39:58 (1774888798) [ 1886.457911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1886.499449] LNet: 81568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.511967] LNetError: 81568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.583490] LNet: Removed LNI 192.168.202.6@tcp [ 1887.620591] LNetError: 81568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.663371] LNet: Removed LNI 192.168.202.6@tcp1 [ 1887.700789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.786432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1887.847285] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1888.780606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1888.855881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1889.072214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.113149] LNet: 81715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.120991] LNetError: 81715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.153439] LNet: Removed LNI 192.168.202.6@tcp [ 1889.197482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.284623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1889.321100] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1890.347898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1890.537523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1891.624151] LNet: Removed LNI 192.168.202.6@tcp [ 1891.647794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.747373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1891.785233] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1893.078748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1893.159797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1893.393365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.419733] LNet: 82008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.426151] LNet: 82008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1893.433363] LNetError: 82008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.439163] LNetError: 82008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1894.508376] LNet: Removed LNI 192.168.202.6@tcp [ 1894.611413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.729632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1894.799717] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1895.921938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1896.196718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1897.255675] LNet: Removed LNI 192.168.202.6@tcp [ 1897.296826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1897.424125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1897.473458] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1898.420437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1898.531181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1899.915333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1899.966290] LNet: 82310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.978602] LNet: 82310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1900.010408] LNetError: 82310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1900.019977] LNetError: 82310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1901.093585] LNet: Removed LNI 192.168.202.6@tcp [ 1901.125884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.209823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1901.232754] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1902.161349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1902.596804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1903.719238] LNet: Removed LNI 192.168.202.6@tcp [ 1903.801337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.959810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1904.014843] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1905.197953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1905.359225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1905.714915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1906.806244] LNet: Removed LNI 192.168.202.6@tcp [ 1906.840162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.947346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1906.998643] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1908.271045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1908.508387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.522191] LNet: 82748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.526218] LNet: 82748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1908.531309] LNetError: 82748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.534662] LNetError: 82748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1908.550685] LNet: Removed LNI 192.168.202.6@tcp [ 1908.577453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.646734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1908.712261] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1909.543429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1909.672232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1911.128643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.240521] LNet: Removed LNI 192.168.202.6@tcp [ 1912.295799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.429932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1912.459486] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1913.416654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1913.658167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.726389] LNet: Removed LNI 192.168.202.6@tcp [ 1914.796340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1914.867550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1914.907295] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1915.816970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1915.940113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1916.319190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.424201] LNet: Removed LNI 192.168.202.6@tcp [ 1917.462373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.525258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1917.595209] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1918.601962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 1918.902634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.980536] LNet: Removed LNI 192.168.202.6@tcp [ 1920.031461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.077204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1920.100362] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1920.687191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1920.735499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.757525] LNet: Removed LNI 192.168.202.6@tcp [ 1920.773130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.810365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1920.832841] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1921.424749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1929.776857] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:40:42 (1774888842) [ 1930.073869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1930.134681] LNet: 84180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.138809] LNet: 84180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1930.156376] LNetError: 84180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.169659] LNetError: 84180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1930.208524] LNet: Removed LNI 192.168.202.6@tcp [ 1930.269869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.388730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1930.421506] LNet: Added LNI 192.168.202.6@tcp231 [8/256/0/180] [ 1932.377782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.439910] LNet: Removed LNI 192.168.202.6@tcp231 [ 1932.459474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.540168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1932.580670] LNet: Added LNI 192.168.202.6@tcp231 [8/256/0/180] [ 1932.590464] LNet: 84326:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1933.253796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1933.292840] LNet: Removed LNI 192.168.202.6@tcp231 [ 1933.334769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1933.382479] LNet: Added LNI 192.168.202.6@tcp231 [8/256/0/180] [ 1935.263824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1935.293814] LNet: Removed LNI 192.168.202.6@tcp231 [ 1935.309319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1935.376680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1935.401935] LNet: Added LNI 192.168.202.6@tcp231 [8/256/0/0] [ 1935.409613] LNet: 84479:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1935.933280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1935.972935] LNet: Removed LNI 192.168.202.6@tcp231 [ 1935.995537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1936.031267] LNet: Added LNI 192.168.202.6@tcp231 [8/256/0/0] [ 1937.480891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.526613] LNet: Removed LNI 192.168.202.6@tcp231 [ 1937.550538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.655683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1937.688953] LNet: Added LNI 192.168.202.6@tcp231 [8/256/0/60] [ 1937.699958] LNet: 84630:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1938.198457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1938.235647] LNet: Removed LNI 192.168.202.6@tcp231 [ 1938.266844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1938.296109] LNet: Added LNI 192.168.202.6@tcp231 [8/256/0/60] [ 1939.515691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1939.545533] LNet: Removed LNI 192.168.202.6@tcp231 [ 1939.560747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.624910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1939.678065] LNet: Added LNI 192.168.202.6@tcp231 [8/256/0/180] [ 1940.322511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1940.374416] LNet: Removed LNI 192.168.202.6@tcp231 [ 1940.391986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1940.426182] LNet: Added LNI 192.168.202.6@tcp231 [8/256/0/180] [ 1950.429951] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:41:03 (1774888863) [ 1950.712688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1950.769100] LNet: Removed LNI 192.168.202.6@tcp231 [ 1950.841534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.911622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1950.934024] LNet: Added LNI 192.168.202.6@tcp232 [8/256/0/180] [ 1952.084431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1952.146902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1952.183343] LNet: Removed LNI 192.168.202.6@tcp232 [ 1952.201232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1952.234854] LNet: Added LNI 192.168.202.6@tcp232 [8/256/0/180] [ 1953.296812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1953.345780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1953.386307] LNet: Removed LNI 192.168.202.6@tcp232 [ 1953.403901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1953.453091] LNet: Added LNI 192.168.202.6@tcp232 [8/256/0/180] [ 1954.240878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1954.281452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1954.312773] LNet: Removed LNI 192.168.202.6@tcp232 [ 1954.337469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1954.383787] LNet: Added LNI 192.168.202.6@tcp232 [8/256/0/180] [ 1955.387108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1955.424783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1955.462046] LNet: Removed LNI 192.168.202.6@tcp232 [ 1955.484902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1962.122473] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:41:15 (1774888875) [ 1962.886987] LNet: 86619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.889819] LNet: 86619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1963.475973] Key type .llcrypt unregistered [ 1963.480219] Key type ._llcrypt unregistered [ 1970.426054] Key type ._llcrypt registered [ 1970.430665] Key type .llcrypt registered [ 1970.725113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1970.745739] alg: No test for adler32 (adler32-zlib) [ 1971.701138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1971.753647] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 1981.101291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 1993.026607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.092165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1993.141940] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 1994.681233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_interrupt -r 2 [ 1994.748262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_interrupt -r 2 [ 1994.822764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_interrupt -r 2 [ 1994.893383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_interrupt -r 2 [ 1994.953163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_interrupt -r 2 [ 1995.032749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_interrupt -r 2 [ 1995.121151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_interrupt -r 2 [ 1995.191359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_interrupt -r 2 [ 1995.261474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.6@tcp 192.168.202.106@tcp [ 1995.316806] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1995.327022] LNet: 1 local NIs in recovery (showing 1): 192.168.202.6@tcp [ 1995.690928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1995.719378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1995.759489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1995.786833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1995.814893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1995.841178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1995.881762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1995.923453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1995.954766] LNet: 87283:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1995.980469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1996.020709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1996.087356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1996.117668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1996.155576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1996.204800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1996.664808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_dropped -r 2 [ 1996.716832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_dropped -r 2 [ 1996.761250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_dropped -r 2 [ 1996.799364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_dropped -r 2 [ 1996.824212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_dropped -r 2 [ 1996.871177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_dropped -r 2 [ 1996.915650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_dropped -r 2 [ 1996.964776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_dropped -r 2 [ 1996.999229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.6@tcp 192.168.202.106@tcp [ 1997.023830] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1997.348312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1997.373489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1997.405130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1997.431361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1997.453234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1997.493386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1997.515749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1997.544278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1997.566316] LNet: 87403:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1997.590687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1997.625090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1997.651905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1997.693828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1997.734945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1997.760194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1998.258307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_aborted -r 2 [ 1998.300297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_aborted -r 2 [ 1998.332871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_aborted -r 2 [ 1998.368402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_aborted -r 2 [ 1998.395718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_aborted -r 2 [ 1998.443841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_aborted -r 2 [ 1998.488479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_aborted -r 2 [ 1998.531321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_aborted -r 2 [ 1998.563808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.6@tcp 192.168.202.106@tcp [ 1998.581380] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 1998.939853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1998.966912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1999.007986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1999.037964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1999.066457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1999.150041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1999.212398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1999.250797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1999.266304] LNet: 87523:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1999.298327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1999.348704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1999.395782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1999.423266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1999.451143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1999.488924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1999.998448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_no_route -r 2 [ 2000.033926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_no_route -r 2 [ 2000.064568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_no_route -r 2 [ 2000.107889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_no_route -r 2 [ 2000.148480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_no_route -r 2 [ 2000.193500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_no_route -r 2 [ 2000.270177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_no_route -r 2 [ 2000.345708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_no_route -r 2 [ 2000.403797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.6@tcp 192.168.202.106@tcp [ 2000.734819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2000.776798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2000.801232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2000.825949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2000.861391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2000.900810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2000.946466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2000.981739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2001.031873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2001.076626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2001.103309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2001.137358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2001.175761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2001.215097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2001.866358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e local_timeout -r 2 [ 2001.939194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e local_timeout -r 2 [ 2002.032129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e local_timeout -r 2 [ 2002.118656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e local_timeout -r 2 [ 2002.199689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e local_timeout -r 2 [ 2002.251686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e local_timeout -r 2 [ 2002.283566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e local_timeout -r 2 [ 2002.316752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e local_timeout -r 2 [ 2002.381579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.6@tcp 192.168.202.106@tcp [ 2002.396350] LNet: There was an unexpected network error while writing to 192.168.202.106: rc = -22 [ 2002.407453] LNet: Skipped 1 previous similar message [ 2002.744375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2002.783937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2002.857500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2002.927794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2002.976592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2003.042502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2003.103953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2003.173368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2003.216433] LNet: 87764:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2003.233176] LNet: 87764:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 2003.264376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2003.316892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2003.377885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2003.416962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2003.466295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2003.515404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2004.213237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp -m GET -e remote_dropped -r 2 [ 2004.252901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.106@tcp1 -m GET -e remote_dropped -r 2 [ 2004.293633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp -m GET -e remote_dropped -r 2 [ 2004.345354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp -d 192.168.202.6@tcp1 -m GET -e remote_dropped -r 2 [ 2004.414799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp -m GET -e remote_dropped -r 2 [ 2004.470459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.106@tcp1 -m GET -e remote_dropped -r 2 [ 2004.551736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp -m GET -e remote_dropped -r 2 [ 2004.649176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.6@tcp1 -d 192.168.202.6@tcp1 -m GET -e remote_dropped -r 2 [ 2004.743970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.6@tcp 192.168.202.106@tcp [ 2004.788305] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.106@tcp [ 2005.112649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2005.151412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2005.201828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2005.256580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2005.324757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2005.382740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2005.426502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2005.480421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2005.521762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2005.556537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2005.620092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2005.670241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2005.761858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2005.825021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2012.545172] LNet: 87958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.554893] LNetError: 87958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2012.567647] LNet: Removed LNI 192.168.202.6@tcp [ 2012.583955] LNet: Removed LNI 192.168.202.6@tcp1 [ 2013.219715] Key type .llcrypt unregistered [ 2013.230220] Key type ._llcrypt unregistered [ 2033.375683] Key type ._llcrypt registered [ 2033.378139] Key type .llcrypt registered [ 2034.755568] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:42:27 (1774888947) [ 2035.420327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2035.444674] alg: No test for adler32 (adler32-zlib) [ 2036.390557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.437612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2036.459904] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2037.245610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2037.272911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.6@tcp [ 2037.293498] LNetError: 88937:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.202.6@tcp to recovery [ 2037.396701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.6@tcp [ 2037.535741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2045.212341] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:42:38 (1774888958) [ 2045.632557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2045.675428] LNet: 89527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.687852] LNetError: 89527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.712469] LNet: Removed LNI 192.168.202.6@tcp [ 2045.745483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.817302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2045.877918] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2046.622635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2046.666122] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 2047.825317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.6@tcp [ 2047.841702] LNetError: 89793:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.202.6@tcp to recovery [ 2048.864387] LNet: 1 local NIs in recovery (showing 1): 192.168.202.6@tcp [ 2048.866814] LNet: There was an unexpected network error while writing to 192.168.202.6: rc = -22 [ 2049.147119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.6@tcp [ 2058.666069] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:42:51 (1774888971) [ 2060.687362] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg206-server [ 2062.645381] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:42:55 (1774888975) [ 2065.167323] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg206-server [ 2067.643354] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:43:00 (1774888980) [ 2070.180733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg206-server [ 2072.800889] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:43:04 (1774888984) [ 2072.904466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2072.944885] LNet: 91417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.954370] LNetError: 91417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2072.963502] LNet: Removed LNI 192.168.202.6@tcp [ 2072.986808] LNet: Removed LNI 192.168.202.6@tcp1 [ 2073.013157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2073.094988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2073.124975] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2073.835923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2073.965879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2074.073630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2074.230617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2074.327063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2074.432672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2081.862588] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:43:14 (1774888994) [ 2082.012571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2082.046968] LNet: 92144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.059354] LNetError: 92144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2082.071122] LNetError: 92144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2082.092558] LNet: Removed LNI 192.168.202.6@tcp [ 2082.109321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.201937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2082.230702] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2089.806294] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:43:22 (1774889002) [ 2090.013481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2090.034713] LNet: 92854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.042604] LNetError: 92854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.078873] LNet: Removed LNI 192.168.202.6@tcp [ 2090.112712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.185444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2090.227497] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2098.292936] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:43:31 (1774889011) [ 2099.774855] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2101.286419] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:43:34 (1774889014) [ 2102.145755] LNet: 93890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.158987] LNetError: 93890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.179618] LNet: Removed LNI 192.168.202.6@tcp [ 2102.744864] Key type .llcrypt unregistered [ 2102.747368] Key type ._llcrypt unregistered [ 2110.404616] Key type ._llcrypt registered [ 2110.411827] Key type .llcrypt registered [ 2110.856500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2110.881640] alg: No test for adler32 (adler32-zlib) [ 2111.830255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2111.899972] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2121.148852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 2127.934977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.106@tcp [ 2128.397600] LNet: 94308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.407406] LNetError: 94308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2128.429079] LNet: Removed LNI 192.168.202.6@tcp [ 2128.933860] Key type .llcrypt unregistered [ 2128.938448] Key type ._llcrypt unregistered [ 2142.061382] Key type ._llcrypt registered [ 2142.063914] Key type .llcrypt registered [ 2143.354738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2145.073375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2146.866770] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:44:19 (1774889059) [ 2147.317453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2147.334860] alg: No test for adler32 (adler32-zlib) [ 2148.347373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.491811] LNet: 95395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.444470] Key type .llcrypt unregistered [ 2149.449387] Key type ._llcrypt unregistered [ 2156.441737] Key type ._llcrypt registered [ 2156.445320] Key type .llcrypt registered [ 2156.769368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2156.789210] alg: No test for adler32 (adler32-zlib) [ 2157.783878] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2157.787610] LNet: 95651:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 2157.794647] LNet: 95651:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2157.870077] LNetError: 95663:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.8@tcp is being used as a gateway but routing feature is not turned on [ 2157.888506] LNetError: 95663:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.8@tcp has gone from up to down [ 2158.588533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.605515] LNet: 95710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2158.614726] LNetError: 95710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2158.635779] LNet: Removed LNI 192.168.202.6@tcp [ 2158.674145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.794470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2158.830128] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2158.835449] LNet: 95717:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2158.842862] LNet: 95717:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 2158.855873] LNetError: 95717:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.8@tcp is a gateway. Can not delete it [ 2158.861541] LNetError: 95717:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.202.8@tcp is a router. Can not be deleted [ 2158.871754] LNet: 95717:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2158.890473] LNetError: 95714:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.202.9@tcp is being used as a gateway but routing feature is not turned on [ 2158.920045] LNetError: 95714:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 2158.938697] LNetError: 95714:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.9@tcp has gone from up to down [ 2158.953447] LNetError: 95714:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 2166.698151] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:44:39 (1774889079) [ 2168.798825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg206-server [ 2170.445142] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:44:43 (1774889083) [ 2172.151273] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg206-server [ 2173.586772] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:44:46 (1774889086) [ 2175.346407] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 2177.175305] Key type lgssc registered [ 2178.493383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2198.983247] Key type lgssc unregistered [ 2199.256259] LNet: 98139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.262711] LNetError: 98139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.302634] LNet: Removed LNI 192.168.202.6@tcp [ 2199.896969] Key type .llcrypt unregistered [ 2199.898710] Key type ._llcrypt unregistered [ 2216.660779] Key type ._llcrypt registered [ 2216.665794] Key type .llcrypt registered [ 2218.119594] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:45:30 (1774889130) [ 2218.726808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2218.748538] alg: No test for adler32 (adler32-zlib) [ 2219.693655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.747369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2219.779205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2219.821272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2219.855969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2225.863745] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:45:39 (1774889139) [ 2226.257407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.285593] LNet: 99545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.306112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.404334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2226.431485] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2227.318710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.334879] LNet: 99689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.342881] LNetError: 99689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.365864] LNet: Removed LNI 192.168.202.6@tcp [ 2227.394561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.444512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2227.467601] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2227.823474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.851305] LNet: Removed LNI 192.168.202.6@tcp [ 2227.867607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.906587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2227.924849] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2234.101852] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:45:46 (1774889146) [ 2234.488764] LNet: 100309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2234.511866] LNet: 100309:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2234.527875] LNetError: 100309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2234.544780] LNetError: 100309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2234.559843] LNet: Removed LNI 192.168.202.6@tcp [ 2235.153561] Key type .llcrypt unregistered [ 2235.156206] Key type ._llcrypt unregistered [ 2241.940469] Key type ._llcrypt registered [ 2241.942639] Key type .llcrypt registered [ 2242.122725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2242.131693] alg: No test for adler32 (adler32-zlib) [ 2243.040641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2243.119358] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2243.124902] LNet: *** cfs_fail_loc=e003, val=100*** [ 2243.804365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 2243.827409] LNet: *** cfs_fail_loc=e003, val=100*** [ 2243.831894] LNet: Skipped 2 previous similar messages [ 2243.842717] Ping reply received with mapping: [ 2243.846619] found metadata reply NI[192.168.202.6@tcp] [ 2243.856188] 06 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2243.862788] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2243.865577] 5a 5a 5a 5a 5a 5a 5a 5a [ 2244.039914] LNet: 100706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.053977] LNetError: 100706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2244.072947] LNet: Removed LNI 192.168.202.6@tcp [ 2244.784459] Key type .llcrypt unregistered [ 2244.786351] Key type ._llcrypt unregistered [ 2256.294502] Key type ._llcrypt registered [ 2256.296573] Key type .llcrypt registered [ 2257.323690] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:46:10 (1774889170) [ 2258.052330] Key type .llcrypt unregistered [ 2258.054495] Key type ._llcrypt unregistered [ 2264.621529] Key type ._llcrypt registered [ 2264.624241] Key type .llcrypt registered [ 2264.876766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2264.911254] alg: No test for adler32 (adler32-zlib) [ 2265.965762] LNetError: 101751:0:(api-ni.c:2734:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2266.305787] Key type .llcrypt unregistered [ 2266.308940] Key type ._llcrypt unregistered [ 2271.076759] Key type ._llcrypt registered [ 2271.078793] Key type .llcrypt registered [ 2272.513071] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:46:25 (1774889185) [ 2273.102398] Key type .llcrypt unregistered [ 2273.104406] Key type ._llcrypt unregistered [ 2279.601212] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2279.608616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2282.337565] Key type ._llcrypt registered [ 2282.340538] Key type .llcrypt registered [ 2282.584644] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2282.604739] alg: No test for adler32 (adler32-zlib) [ 2283.540412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2283.606050] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2292.913907] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:46:46 (1774889206) [ 2293.117434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.150197] LNet: 103533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.154940] LNetError: 103533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2293.168945] LNet: Removed LNI 10.1.2.3@tcp [ 2293.189670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.867831] Key type .llcrypt unregistered [ 2293.871143] Key type ._llcrypt unregistered [ 2299.555289] Key type ._llcrypt registered [ 2299.557693] Key type .llcrypt registered [ 2299.694151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2299.707789] alg: No test for adler32 (adler32-zlib) [ 2300.607754] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2300.677482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 2300.833447] LNet: 103821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.840468] LNetError: 103821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2300.869034] LNet: Removed LNI 192.168.202.6@tcp [ 2301.298526] Key type .llcrypt unregistered [ 2301.300553] Key type ._llcrypt unregistered [ 2312.055917] Key type ._llcrypt registered [ 2312.060497] Key type .llcrypt registered [ 2313.473427] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:47:06 (1774889226) [ 2314.272739] Key type .llcrypt unregistered [ 2314.278763] Key type ._llcrypt unregistered [ 2320.306933] Key type ._llcrypt registered [ 2320.315799] Key type .llcrypt registered [ 2320.544870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2320.578216] alg: No test for adler32 (adler32-zlib) [ 2326.922574] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:47:20 (1774889240) [ 2327.271950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2327.349902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2327.461206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2327.502145] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2328.361910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2328.429434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2328.476728] LNet: Removed LNI 192.168.202.6@tcp [ 2328.663723] LNet: 105564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2328.944548] Key type .llcrypt unregistered [ 2328.947996] Key type ._llcrypt unregistered [ 2340.190421] Key type ._llcrypt registered [ 2340.192507] Key type .llcrypt registered [ 2341.379506] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:47:34 (1774889254) [ 2341.809137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2341.826263] alg: No test for adler32 (adler32-zlib) [ 2342.781590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.831318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2342.851558] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2343.600477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 2343.624557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.6@tcp [ 2343.638499] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2343.642105] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2343.644830] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2343, 0, 0 [ 2343.649425] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2343.651884] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2343.657964] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2343.661341] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2343, 0, 0 [ 2343.665352] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2343.667734] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2343.670945] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2343.673944] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2343, 0, 2393 [ 2343.678604] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2343.680545] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2343.683598] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2343.686469] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2343, 0, 2393 [ 2343.692125] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2349.368271] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:47:42 (1774889262) [ 2349.833667] LNet: 107103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.837461] LNetError: 107103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2349.850444] LNet: Removed LNI 192.168.202.6@tcp [ 2350.336576] Key type .llcrypt unregistered [ 2350.341976] Key type ._llcrypt unregistered [ 2356.940876] Key type ._llcrypt registered [ 2356.942990] Key type .llcrypt registered [ 2357.160167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2357.177366] alg: No test for adler32 (adler32-zlib) [ 2358.125808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2358.182829] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2367.293878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 2377.431595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2377.491649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2377.512585] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 2378.275157] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2378.400883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2378.412981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2378.427180] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2380.898783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2380.931763] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2390.310318] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:48:23 (1774889303) [ 2390.670787] LNet: 108605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.680882] LNetError: 108605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.690379] LNet: Removed LNI 192.168.202.6@tcp [ 2391.711251] LNet: Removed LNI 192.168.202.6@tcp1 [ 2391.715477] LNetError: 108605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2391.721879] LNetError: 108605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2391.751839] LNet: Removed LNI 10.1.2.3@tcp99 [ 2392.374785] Key type .llcrypt unregistered [ 2392.379654] Key type ._llcrypt unregistered [ 2397.661039] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2397.670270] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2400.367260] Key type ._llcrypt registered [ 2400.369814] Key type .llcrypt registered [ 2400.623671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2400.637447] alg: No test for adler32 (adler32-zlib) [ 2401.564925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.611996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2401.658564] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2402.296374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2402.312603] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2404.020723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2404.061270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2404.090783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2404.108768] LNetError: 109349:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2404.153209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2404.207377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2404.616187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2404.677233] LNet: 109369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2404.682691] LNetError: 109369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2405.727198] LNet: Removed LNI 192.168.202.6@tcp [ 2405.733604] LNetError: 109369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2405.757281] LNet: Removed LNI 10.1.2.3@tcp [ 2406.288531] Key type .llcrypt unregistered [ 2406.291094] Key type ._llcrypt unregistered [ 2416.914949] Key type ._llcrypt registered [ 2416.916483] Key type .llcrypt registered [ 2418.098271] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:48:51 (1774889331) [ 2418.507345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2418.534392] alg: No test for adler32 (adler32-zlib) [ 2419.501782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2419.559312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2419.585063] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2420.272964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.6@tcp [ 2420.362468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg206-client@tcp [ 2425.784832] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:48:59 (1774889339) [ 2426.122858] LNet: 110903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.140596] LNetError: 110903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.168791] LNet: Removed LNI 192.168.202.6@tcp [ 2426.801631] Key type .llcrypt unregistered [ 2426.807303] Key type ._llcrypt unregistered [ 2432.074941] Key type ._llcrypt registered [ 2432.076944] Key type .llcrypt registered [ 2432.199913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2432.214740] alg: No test for adler32 (adler32-zlib) [ 2433.116654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2433.141230] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2433.190154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2439.089255] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:49:12 (1774889352) [ 2440.527714] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2441.832470] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:49:15 (1774889355) [ 2443.046475] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2444.327301] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:49:17 (1774889357) [ 2444.562340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2444.583937] LNet: 112323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.588566] LNetError: 112323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.602535] LNet: Removed LNI 192.168.202.6@tcp [ 2444.615989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2444.670909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2444.697337] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2445.291940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2445.334668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2451.139853] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:49:24 (1774889364) [ 2451.371959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2451.388899] LNet: 113033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.394703] LNetError: 113033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2451.409737] LNet: Removed LNI 192.168.202.6@tcp [ 2451.425883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.488356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2451.515932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2452.182435] Key type .llcrypt unregistered [ 2452.185405] Key type ._llcrypt unregistered [ 2463.787889] Key type ._llcrypt registered [ 2463.790328] Key type .llcrypt registered [ 2464.936102] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:49:38 (1774889378) [ 2465.318406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2465.336774] alg: No test for adler32 (adler32-zlib) [ 2466.347270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2466.387708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2466.412327] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2466.925258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2466.980044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2472.252621] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:49:45 (1774889385) [ 2472.324968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2472.345113] LNet: 114583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.351371] LNetError: 114583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2472.360692] LNet: Removed LNI 192.168.202.6@tcp [ 2472.386180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2472.437951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2472.473980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2477.781447] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:49:51 (1774889391) [ 2477.956576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2477.975168] LNet: 115169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2477.996498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2478.049718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2478.356905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2478.384043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2478.424768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2478.733719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2478.752101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2478.799729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2478.829343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2478.890706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2479.131940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2479.153095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2479.192548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2479.431500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2479.458509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2485.089128] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:49:58 (1774889398) [ 2485.484863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2485.510810] LNet: 115802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2485.520285] LNet: 115802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2485.555589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2485.639942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2485.684579] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2486.222409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2486.262899] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2487.001950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2487.073518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2487.313612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2487.336702] LNetError: 116082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2487.342993] LNet: Removed LNI 192.168.202.6@tcp [ 2487.352712] LNet: Removed LNI 192.168.202.6@tcp2 [ 2487.370559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2487.422639] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2487.426587] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2487.433664] LNet: 116088:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2487.923704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2488.126907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2488.151036] LNet: 116125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2488.160555] LNet: 116125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2489.183337] LNet: Removed LNI 192.168.202.6@tcp [ 2489.186887] LNetError: 116125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2489.200505] LNetError: 116125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2489.216804] LNet: Removed LNI 192.168.202.6@tcp2 [ 2489.246241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2489.319633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2489.354843] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2490.232464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2490.264038] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2491.173431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2491.230786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2491.315496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2491.556789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2491.576920] LNetError: 116410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2492.639903] LNet: Removed LNI 192.168.202.6@tcp [ 2493.673419] LNet: Removed LNI 192.168.202.6@tcp2 [ 2493.716767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2493.816820] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2493.821756] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2493.826154] LNet: 116416:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2494.298709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2494.476588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2494.489448] LNet: 116453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2494.494334] LNet: 116453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2494.500711] LNet: Removed LNI 192.168.202.6@tcp [ 2495.535750] LNet: Removed LNI 192.168.202.6@tcp2 [ 2495.553071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2495.625712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2495.646477] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2496.230583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2496.254644] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2496.923634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2496.971332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2497.141705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2497.156912] LNetError: 116733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2497.160570] LNetError: 116733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2498.207234] LNet: Removed LNI 192.168.202.6@tcp [ 2498.218567] LNet: Removed LNI 192.168.202.6@tcp2 [ 2498.253516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2498.339499] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2498.343630] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2498.350732] LNet: 116739:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2498.793496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2498.970977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2499.999654] LNet: Removed LNI 192.168.202.6@tcp [ 2500.021018] LNet: Removed LNI 192.168.202.6@tcp2 [ 2500.042654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2500.113897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2500.135321] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2500.620427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2500.636641] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2501.286698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2501.322124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2501.379814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2501.597528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2502.623197] LNet: Removed LNI 192.168.202.6@tcp [ 2503.667324] LNet: Removed LNI 192.168.202.6@tcp2 [ 2503.726105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2503.842544] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2503.851612] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2503.861285] LNet: 117067:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2504.429837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2504.663735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2504.698983] LNet: 117104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2504.703438] LNet: 117104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2504.710074] LNet: Removed LNI 192.168.202.6@tcp [ 2505.767678] LNet: Removed LNI 192.168.202.6@tcp2 [ 2505.790598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2505.865877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2505.893239] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2506.707808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2506.727699] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2507.501498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2507.534633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2507.592153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2507.927805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2507.957520] LNetError: 117401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2507.964908] LNetError: 117401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2507.977346] LNet: Removed LNI 192.168.202.6@tcp [ 2507.989222] LNet: Removed LNI 192.168.202.6@tcp2 [ 2508.025718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2508.095043] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2508.099389] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2508.101970] LNet: 117407:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2508.618958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2508.904283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2509.988789] LNet: Removed LNI 192.168.202.6@tcp [ 2509.999902] LNet: Removed LNI 192.168.202.6@tcp2 [ 2510.054502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2510.132761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2510.153708] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2510.737960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2510.758779] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2511.504222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2511.532926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2511.560332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2511.605746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2511.838780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2512.863701] LNet: Removed LNI 192.168.202.6@tcp [ 2513.893945] LNet: Removed LNI 192.168.202.6@tcp2 [ 2513.920575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2513.975862] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2513.987043] LNet: Added LNI 192.168.202.6@tcp2 [8/256/0/180] [ 2514.313719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2519.821466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2521.297863] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:50:34 (1774889434) [ 2521.591394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2521.611821] LNet: 118527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.619855] LNet: 118527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2521.629914] LNet: Removed LNI 192.168.202.6@tcp [ 2522.662395] LNet: Removed LNI 192.168.202.6@tcp2 [ 2522.717646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.937659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2522.955541] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2523.478692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2523.492383] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2524.137887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2524.153445] LNetError: 118800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.159480] LNetError: 118800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2524.166328] LNet: Removed LNI 192.168.202.6@tcp [ 2524.177884] LNet: Removed LNI 10.1.2.3@tcp [ 2524.194223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2524.233601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2524.255604] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2524.259867] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2524.719533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2524.748389] LNet: Removed LNI 192.168.202.6@tcp [ 2524.761299] LNet: Removed LNI 10.1.2.3@tcp [ 2530.133573] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:50:43 (1774889443) [ 2530.259705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2530.304165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2530.441385] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2530.706705] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2530.716169] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2530.724080] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2533.086699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2533.113757] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2533.882731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2533.917887] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2536.066283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.6@tcp [ 2536.123113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2536.167334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.6@tcp -m PUT -e local_timeout -r 1 [ 2536.229875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2536.279920] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2536.284271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2536.487744] LNetError: 119461:0:(lib-move.c:4960:lnet_parse()) 192.168.202.6@tcp, src 192.168.202.6@tcp: Dropping PUT (error -108 looking up sender) [ 2538.527164] LNet: Removed LNI 192.168.202.6@tcp [ 2538.539355] LNet: Removed LNI 10.1.2.3@tcp [ 2538.940229] Key type .llcrypt unregistered [ 2538.943712] Key type ._llcrypt unregistered [ 2549.554666] Key type ._llcrypt registered [ 2549.558698] Key type .llcrypt registered [ 2550.801715] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:51:03 (1774889463) [ 2551.154518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2551.172558] alg: No test for adler32 (adler32-zlib) [ 2552.062396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2552.378990] LNet: 120772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2552.861366] Key type .llcrypt unregistered [ 2552.863225] Key type ._llcrypt unregistered [ 2557.103754] Key type ._llcrypt registered [ 2557.105684] Key type .llcrypt registered [ 2557.277589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2557.289918] alg: No test for adler32 (adler32-zlib) [ 2558.191576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2558.230110] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2564.109628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 2564.253478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2564.675682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.106@tcp [ 2564.861339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2564.882318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2564.904251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2564.927311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2564.961578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2564.983784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2565.006631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2565.032536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2565.045986] LNet: 121222:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2565.058717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2565.082837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2565.106456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2565.127359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2565.149586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2565.177326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2565.680871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2565.874970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2565.901547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2565.924266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2565.947499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2565.970926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2565.992737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2566.018333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2566.042275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2566.052917] LNet: 121339:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2566.068484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2566.089298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2566.114379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2566.130913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2566.159297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2566.181480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2566.711775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.106@tcp [ 2566.897778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2566.916560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2566.938775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2566.957993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2566.985702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2567.014885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2567.043669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2567.068219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2567.080614] LNet: 121456:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2567.092918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2567.130309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2567.155898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2567.183396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2567.207471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2567.225710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2567.562812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2567.710311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2567.726792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2567.743559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2567.757211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2567.783451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2567.802349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2567.824572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2567.845876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2567.866717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2567.883917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2567.906442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2567.931218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2567.957658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2567.982077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2568.183437] LNet: 121620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.189293] LNetError: 121620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2568.198705] LNet: Removed LNI 192.168.202.6@tcp [ 2568.473896] Key type .llcrypt unregistered [ 2568.476256] Key type ._llcrypt unregistered [ 2581.106364] Key type ._llcrypt registered [ 2581.107693] Key type .llcrypt registered [ 2582.174759] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:51:35 (1774889495) [ 2582.737738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2582.757082] alg: No test for adler32 (adler32-zlib) [ 2583.703551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2584.150785] LNet: 122507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2584.636368] Key type .llcrypt unregistered [ 2584.637839] Key type ._llcrypt unregistered [ 2590.626783] Key type ._llcrypt registered [ 2590.629315] Key type .llcrypt registered [ 2590.776941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2590.787561] alg: No test for adler32 (adler32-zlib) [ 2591.733210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2591.777398] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2598.323882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.106@tcp [ 2605.036959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2605.067000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2605.083905] LNet: Added LNI 192.168.202.6@tcp1 [8/256/0/180] [ 2605.812915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2606.388413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.106@tcp [ 2606.607738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2606.631580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2606.667592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2606.698633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2606.733484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2606.768958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2606.798749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2606.840694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2606.864830] LNet: 123159:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2606.879307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2606.914783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2606.949642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2606.995626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2607.024564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2607.056367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2607.616333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2607.849355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2607.905364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2607.951729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2607.981827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2608.002461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2608.036144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2608.083718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2608.113237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2608.142125] LNet: 123276:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2608.159517] LNet: 2 peer NIs in recovery (showing 2): 192.168.202.106@tcp, 192.168.202.106@tcp1 [ 2608.173952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2608.206966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2608.266043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2608.300335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2608.333640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2608.368991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2608.892448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.106@tcp [ 2609.081957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2609.107880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2609.129889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2609.171431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2609.194905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2609.222582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2609.246526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2609.270349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2609.287956] LNet: 123394:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2609.301596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2609.328928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2609.358667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2609.391961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2609.422821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2609.459669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2610.158417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2610.379938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2610.400401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2610.425635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2610.450638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2610.477892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2610.504606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2610.523421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2610.540284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2610.559789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2610.586239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2610.606655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2610.634527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2610.667604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2610.691721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2614.779628] LNet: 123592:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2614.790320] LNetError: 123592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2614.801048] LNet: Removed LNI 192.168.202.6@tcp [ 2614.815798] LNet: Removed LNI 192.168.202.6@tcp1 [ 2615.255933] Key type .llcrypt unregistered [ 2615.258972] Key type ._llcrypt unregistered [ 2629.241796] Key type ._llcrypt registered [ 2629.243754] Key type .llcrypt registered [ 2630.303481] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:52:23 (1774889543) [ 2636.173322] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:52:29 (1774889549) [ 2642.398263] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:52:35 (1774889555) [ 2648.267709] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:52:41 (1774889561) [ 2654.654647] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:52:47 (1774889567) [ 2661.466368] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:52:54 (1774889574) [ 2668.502808] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:53:01 (1774889581) [ 2675.119790] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:53:08 (1774889588) [ 2681.915406] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:53:14 (1774889594) [ 2687.737684] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:53:21 (1774889601) [ 2693.615860] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:53:26 (1774889606) [ 2700.682142] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:53:33 (1774889613) [ 2706.948958] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:53:40 (1774889620) [ 2712.173226] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:53:45 (1774889625) [ 2717.102567] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:53:50 (1774889630) [ 2721.567940] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:53:55 (1774889635) [ 2726.455327] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:53:59 (1774889639) [ 2731.920852] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:54:05 (1774889645) [ 2732.262760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2732.274763] alg: No test for adler32 (adler32-zlib) [ 2733.186765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2733.336713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2733.357927] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2734.339607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2734.369340] LNet: 134298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2734.375669] LNetError: 134298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2734.387185] LNet: Removed LNI 192.168.202.6@tcp [ 2734.403690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2734.442576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2734.457768] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2740.170682] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:54:13 (1774889653) [ 2746.139790] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:54:19 (1774889659) [ 2751.050402] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:54:24 (1774889664) [ 2756.911796] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:54:29 (1774889669) [ 2762.384390] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:54:35 (1774889675) [ 2767.752739] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:54:41 (1774889681) [ 2772.881636] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:54:46 (1774889686) [ 2777.956672] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:54:51 (1774889691) [ 2783.043922] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:54:56 (1774889696) [ 2788.133166] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:55:01 (1774889701) [ 2793.267347] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:55:06 (1774889706) [ 2798.437556] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:55:11 (1774889711) [ 2802.634763] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2494 sec ========== 12:55:16 (1774889716) [ 2802.889654] LNet: 141762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2802.894249] LNetError: 141762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2802.909660] LNet: Removed LNI 192.168.202.6@tcp [ 2803.322835] Key type .llcrypt unregistered [ 2803.324035] Key type ._llcrypt unregistered [ 2811.893515] Key type ._llcrypt registered [ 2811.895288] Key type .llcrypt registered [ 2812.096402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2812.104596] alg: No test for adler32 (adler32-zlib) [ 2813.070171] Lustre: Lustre: Build Version: 2.17.50_165_ga4ea30a [ 2813.212706] LNet: Added LNI 192.168.202.6@tcp [8/256/0/180] [ 2814.839286] Key type lgssc registered [ 2815.683460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2859.526356] Lustre: Mounted lustre-client [ 2863.731092] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2874.068211] LustreError: 144218:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8ac885732800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2874.107897] Lustre: Unmounted lustre-client [ 2907.380591] Key type lgssc unregistered [ 2907.558198] LNet: 144701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2907.568634] LNetError: 144701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2907.592648] LNet: Removed LNI 192.168.202.6@tcp [ 2908.166150] Key type .llcrypt unregistered [ 2908.169928] Key type ._llcrypt unregistered