[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461284530 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002317] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006830] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008018] pid_max: default: 32768 minimum: 301 [ 0.009146] LSM: Security Framework initializing [ 0.010064] Yama: becoming mindful. [ 0.011042] SELinux: Initializing. [ 0.012068] *** VALIDATE selinux *** [ 0.020481] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024667] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027106] *** VALIDATE tmpfs *** [ 0.029266] *** VALIDATE proc *** [ 0.030231] *** VALIDATE cgroup *** [ 0.031008] *** VALIDATE cgroup2 *** [ 0.032274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035040] Spectre V2 : User space: Vulnerable [ 0.036006] Speculative Store Bypass: Vulnerable [ 0.039547] debug: unmapping init [mem 0xffffffffa9659000-0xffffffffa9660fff] [ 0.042180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043696] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044023] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048011] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.049996] ... event mask: 000000070000000f [ 0.051264] rcu: Hierarchical SRCU implementation. [ 0.053702] smp: Bringing up secondary CPUs ... [ 0.054558] x86: Booting SMP configuration: [ 0.055028] .... node #0, CPUs: #1 #2 #3 [ 0.062139] smp: Brought up 1 node, 4 CPUs [ 0.064012] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.105369] node 0 deferred pages initialised in 38ms [ 0.109046] devtmpfs: initialized [ 0.110214] x86/mm: Memory block size: 128MB [ 0.113060] gcov: version magic: 0x41383552 [ 0.115322] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.116142] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.117282] pinctrl core: initialized pinctrl subsystem [ 0.118192] [ 0.118943] ************************************************************* [ 0.119011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.120012] ** ** [ 0.121055] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.122010] ** ** [ 0.123011] ** This means that this kernel is built to expose internal ** [ 0.124011] ** IOMMU data structures, which may compromise security on ** [ 0.125013] ** your system. ** [ 0.126016] ** ** [ 0.127016] ** If you see this message and you are not debugging the ** [ 0.128014] ** kernel, report this immediately to your vendor! ** [ 0.129011] ** ** [ 0.130013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131013] ************************************************************* [ 0.132939] NET: Registered protocol family 16 [ 0.133720] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.134054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.135060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.136846] cpuidle: using governor menu [ 0.137729] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.140672] PCI: Using configuration type 1 for base access [ 0.142195] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.152116] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.156031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.161044] cryptd: max_cpu_qlen set to 1000 [ 0.164100] ACPI: Added _OSI(Module Device) [ 0.165011] ACPI: Added _OSI(Processor Device) [ 0.167010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.168012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.173204] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.179641] ACPI: Interpreter enabled [ 0.181078] ACPI: PM: (supports S0 S3 S4 S5) [ 0.183013] ACPI: Using IOAPIC for interrupt routing [ 0.185149] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.188664] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.201187] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.204045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.207022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.210094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.216908] acpiphp: Slot [2] registered [ 0.218159] acpiphp: Slot [5] registered [ 0.220213] acpiphp: Slot [6] registered [ 0.222122] acpiphp: Slot [3] registered [ 0.224105] acpiphp: Slot [4] registered [ 0.225139] acpiphp: Slot [7] registered [ 0.227095] acpiphp: Slot [8] registered [ 0.229104] acpiphp: Slot [9] registered [ 0.230090] acpiphp: Slot [10] registered [ 0.232091] acpiphp: Slot [11] registered [ 0.233111] acpiphp: Slot [12] registered [ 0.235131] acpiphp: Slot [13] registered [ 0.237157] acpiphp: Slot [14] registered [ 0.238173] acpiphp: Slot [15] registered [ 0.240096] acpiphp: Slot [16] registered [ 0.242112] acpiphp: Slot [17] registered [ 0.243267] acpiphp: Slot [18] registered [ 0.245115] acpiphp: Slot [19] registered [ 0.247112] acpiphp: Slot [20] registered [ 0.249083] acpiphp: Slot [21] registered [ 0.250115] acpiphp: Slot [22] registered [ 0.252150] acpiphp: Slot [23] registered [ 0.253167] acpiphp: Slot [24] registered [ 0.256091] acpiphp: Slot [25] registered [ 0.257146] acpiphp: Slot [26] registered [ 0.258183] acpiphp: Slot [27] registered [ 0.260086] acpiphp: Slot [28] registered [ 0.261079] acpiphp: Slot [29] registered [ 0.263092] acpiphp: Slot [30] registered [ 0.264077] acpiphp: Slot [31] registered [ 0.265088] PCI host bridge to bus 0000:00 [ 0.267023] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.269023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.271043] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.273100] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.276029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.278025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.280211] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.283203] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.286497] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.294015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.297474] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.300015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.302017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.304011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.306711] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.308780] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.311034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.314000] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.316951] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.325628] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.329020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.334000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.345019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.354021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.406019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.425783] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.435018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.442024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.463000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.466666] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.467397] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.470528] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.473406] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.481324] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.489420] iommu: Default domain type: Passthrough [ 0.494841] SCSI subsystem initialized [ 0.497193] ACPI: bus type USB registered [ 0.500193] usbcore: registered new interface driver usbfs [ 0.504260] usbcore: registered new interface driver hub [ 0.508310] usbcore: registered new device driver usb [ 0.513410] pps_core: LinuxPPS API ver. 1 registered [ 0.516015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.523144] PTP clock support registered [ 0.527116] EDAC MC: Ver: 3.0.0 [ 0.530286] PCI: Using ACPI for IRQ routing [ 0.545666] NetLabel: Initializing [ 0.548016] NetLabel: domain hash size = 128 [ 0.552017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.556200] NetLabel: unlabeled traffic allowed by default [ 0.560247] vgaarb: loaded [ 0.563081] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.566012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.575230] clocksource: Switched to clocksource kvm-clock [ 0.780046] VFS: Disk quotas dquot_6.6.0 [ 0.781856] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.784046] *** VALIDATE ramfs *** [ 0.784922] *** VALIDATE hugetlbfs *** [ 0.789088] pnp: PnP ACPI init [ 0.791418] pnp: PnP ACPI: found 6 devices [ 0.835365] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.842924] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.847898] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.850388] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.854771] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.858867] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.866925] NET: Registered protocol family 2 [ 0.873026] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.884992] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.890934] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.900336] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.908617] TCP: Hash tables configured (established 65536 bind 65536) [ 0.920118] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.921964] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.923899] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.926053] NET: Registered protocol family 1 [ 0.930946] RPC: Registered named UNIX socket transport module. [ 0.933839] RPC: Registered udp transport module. [ 0.935519] RPC: Registered tcp transport module. [ 0.937290] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.939413] NET: Registered protocol family 44 [ 0.941451] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.943339] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.946407] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.948675] PCI: CLS 0 bytes, default 64 [ 0.950206] Unpacking initramfs... [ 3.705463] debug: unmapping init [mem 0xffff8f00fcc64000-0xffff8f00fffcffff] [ 3.716994] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.728374] software IO TLB: mapped [mem 0x00000000a3e00000-0x00000000a7e00000] (64MB) [ 3.748374] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.142153] Initialise system trusted keyrings [ 5.145818] Key type blacklist registered [ 5.152368] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.186844] zbud: loaded [ 5.196711] *** VALIDATE nfs *** [ 5.200218] *** VALIDATE nfs4 *** [ 5.206623] pstore: using deflate compression [ 5.222866] Platform Keyring initialized [ 5.355973] NET: Registered protocol family 38 [ 5.358069] Key type asymmetric registered [ 5.360237] Asymmetric key parser 'x509' registered [ 5.362609] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.365364] io scheduler mq-deadline registered [ 5.366786] io scheduler kyber registered [ 5.368594] io scheduler bfq registered [ 5.370520] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.373879] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.377465] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.380862] ACPI: Power Button [PWRF] [ 5.387216] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.398189] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.414181] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.442236] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.472363] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.476831] Non-volatile memory driver v1.3 [ 5.478522] Linux agpgart interface v0.103 [ 5.509963] virtio_blk virtio1: [vda] 133944 512-byte logical blocks (68.6 MB/65.4 MiB) [ 5.514078] vda: detected capacity change from 0 to 68579328 [ 5.615964] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.620241] vdb: detected capacity change from 0 to 1073741824 [ 5.634432] libphy: Fixed MDIO Bus: probed [ 5.642537] usbcore: registered new interface driver usbserial_generic [ 5.645193] usbserial: USB Serial support registered for generic [ 5.647631] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.661762] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.665920] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.674896] mousedev: PS/2 mouse device common for all mice [ 5.685735] rtc_cmos 00:05: RTC can wake from S4 [ 5.692442] rtc_cmos 00:05: registered as rtc0 [ 5.695672] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.696190] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.710393] intel_pstate: CPU model not supported [ 5.714460] hid: raw HID events driver (C) Jiri Kosina [ 5.717435] usbcore: registered new interface driver usbhid [ 5.718473] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.723353] usbhid: USB HID core driver [ 5.736272] drop_monitor: Initializing network drop monitor service [ 5.740866] Initializing XFRM netlink socket [ 5.743172] NET: Registered protocol family 10 [ 5.744276] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.749873] Segment Routing with IPv6 [ 5.754203] NET: Registered protocol family 17 [ 5.756463] mpls_gso: MPLS GSO support [ 5.765160] RAS: Correctable Errors collector initialized. [ 5.767534] AVX version of gcm_enc/dec engaged. [ 5.770145] AES CTR mode by8 optimization enabled [ 5.921206] sched_clock: Marking stable (5921095602, 0)->(6956849019, -1035753417) [ 5.934333] registered taskstats version 1 [ 5.937814] Loading compiled-in X.509 certificates [ 5.944514] zswap: loaded using pool lzo/zbud [ 6.038267] Key type big_key registered [ 6.061320] Key type encrypted registered [ 6.064218] ima: No TPM chip found, activating TPM-bypass! [ 6.075179] ima: Allocated hash algorithm: sha1 [ 6.080848] ima: No architecture policies found [ 6.086514] evm: Initialising EVM extended attributes: [ 6.092344] evm: security.selinux [ 6.096578] evm: security.ima [ 6.101341] evm: security.capability [ 6.107017] evm: HMAC attrs: 0x1 [ 6.117833] rtc_cmos 00:05: setting system clock to 2026-04-23 04:39:15 UTC (1776919155) [ 6.140474] debug: unmapping init [mem 0xffffffffaa603000-0xffffffffaa7fffff] [ 6.145353] debug: unmapping init [mem 0xffffffffa9382000-0xffffffffa9658fff] [ 6.159073] Write protecting the kernel read-only data: 28672k [ 6.164220] debug: unmapping init [mem 0xffffffffa7a03000-0xffffffffa7bfffff] [ 6.167756] debug: unmapping init [mem 0xffffffffa8314000-0xffffffffa83fffff] [ 6.264640] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.281722] systemd[1]: Detected virtualization kvm. [ 6.283027] systemd[1]: Detected architecture x86-64. [ 6.286302] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.333367] systemd[1]: No hostname configured. [ 6.336846] systemd[1]: Set hostname to . [ 6.341240] random: systemd: uninitialized urandom read (16 bytes read) [ 6.346735] systemd[1]: Initializing machine ID from random generator. [ 6.488448] random: ln: uninitialized urandom read (6 bytes read) [ 6.792857] random: systemd: uninitialized urandom read (16 bytes read) [ 6.831402] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 6.852788] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 6.887564] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.752848] device-mapper: uevent: version 1.0.3 [ 9.755973] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.744938] virtio_net virtio0 ens2: renamed from eth0 [ 13.570713] scsi host0: ata_piix [ 13.613337] scsi host1: ata_piix [ 13.615110] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 13.623268] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 17.696669] random: crng init done [ 17.698753] random: 7 urandom warning(s) missed due to ratelimiting [ 18.176705] hrtimer: interrupt took 10738920 ns [ 21.282601] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.654234] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.255483] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.061857] SELinux: Disabled at runtime. [ 26.172962] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.189799] systemd[1]: Detected virtualization kvm. [ 26.193254] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 27.768511] systemd[1]: initrd-switch-root.service: Succeeded. [ 27.775074] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 27.817745] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 27.828809] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 27.836088] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 27.861491] systemd[1]: Starting Journal Service... Starting Journal Service... [ 27.909281] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached targ[ 28.363723] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 29.670494] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.119774] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.141073] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.717897] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.814360] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit)[ 35.724747] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 36.942284] NFS: Registering the id_resolver key type [ 36.946272] Key type id_resolver registered [ 36.950402] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg305-client login: [ 85.547339] libcfs: loading out-of-tree module taints kernel. [ 85.570384] Key type ._llcrypt registered [ 85.576100] Key type .llcrypt registered [ 85.925859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 85.933452] alg: No test for adler32 (adler32-zlib) [ 86.983585] Lustre: Lustre: Build Version: 2.17.51_74_g68f2299 [ 87.325476] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 88.951139] Key type lgssc registered [ 89.839567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 162.133922] Lustre: Mounted lustre-client [ 165.053542] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 177.598987] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing check_logdir /tmp/testlogs/ [ 180.739645] Lustre: DEBUG MARKER: oleg305-client.virtnet: executing yml_node [ 183.765875] Lustre: DEBUG MARKER: Client: 2.17.51.74 [ 185.389843] Lustre: DEBUG MARKER: MDS: 2.17.51.74 [ 187.017262] Lustre: DEBUG MARKER: OSS: 2.17.51.74 [ 187.873200] Lustre: lustre-OST0000-osc-ffff8f01500cd800: disconnect after 24s idle [ 188.077849] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Apr 23 00:42:16 EDT 2026 [ 196.028927] LustreError: 7905:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8f01500cd800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 196.052253] Lustre: Unmounted lustre-client [ 207.331604] Key type lgssc unregistered [ 207.489985] LNet: 8348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 207.495847] LNetError: 8348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 207.510688] LNet: Removed LNI 192.168.203.5@tcp [ 207.997060] Key type .llcrypt unregistered [ 208.002484] Key type ._llcrypt unregistered [ 223.007822] Key type ._llcrypt registered [ 223.013730] Key type .llcrypt registered [ 223.120659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 223.136315] alg: No test for adler32 (adler32-zlib) [ 223.979149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 224.027035] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 224.045449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 226.382419] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 226.463258] LNet: 9235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 226.472879] LNetError: 9235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 226.491856] LNet: Removed LNI 192.168.203.5@tcp [ 226.905379] Key type .llcrypt unregistered [ 226.908955] Key type ._llcrypt unregistered [ 229.945795] Key type ._llcrypt registered [ 229.949329] Key type .llcrypt registered [ 230.538095] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:42:59 (1776919379) [ 230.687801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 230.696851] alg: No test for adler32 (adler32-zlib) [ 231.515709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 231.564577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 231.577586] LNet: 9645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 231.593284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 231.627400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 231.640233] LNet: 9652:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 235.126477] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:43:03 (1776919383) [ 235.170894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 235.196184] LNet: 10224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 235.215613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 235.265896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 238.797328] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:43:07 (1776919387) [ 238.938952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 238.959737] LNet: 10813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 238.977995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.018454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 242.871748] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:43:11 (1776919391) [ 242.987338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 243.011528] LNet: 11397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 243.028470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 243.065432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 247.153623] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:43:15 (1776919395) [ 247.266794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.295342] LNet: 11980:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 247.318840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.364855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 247.658758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 251.606786] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:43:20 (1776919400) [ 251.697795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 251.736167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 251.785402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 252.009394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 252.346610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 256.226343] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:43:24 (1776919404) [ 256.355947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 256.379388] LNet: 13167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 256.383901] LNet: 13167:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 256.395933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 256.443622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 260.153666] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:43:28 (1776919408) [ 260.207684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.228721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.272610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 260.298922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 260.495639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 260.517741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 260.713577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 260.737265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 260.981910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 261.009187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 261.221419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 261.243486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 261.465955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 261.485989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 261.679961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 261.702537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 261.971622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 261.993773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 262.241856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 262.266453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 262.477930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 262.503090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 262.738841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 262.770099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 263.007271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 263.046571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 263.375805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 263.407249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 267.545624] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:43:36 (1776919416) [ 267.602966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.635405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.694846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 267.711843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 271.862143] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:43:40 (1776919420) [ 271.922737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.955767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.017705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 272.051842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 275.961591] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:43:44 (1776919424) [ 276.067157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.088672] LNet: 15602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.095326] LNet: 15602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 276.109918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.155308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 276.173901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 280.044165] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:43:48 (1776919428) [ 280.178435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.204600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.257602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 280.289203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 283.941288] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:43:52 (1776919432) [ 284.020845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 284.057802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.106033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 284.132187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 288.452348] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:43:56 (1776919436) [ 288.533585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.597321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.680509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 288.713275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 292.806196] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:44:01 (1776919441) [ 292.874735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.924236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.026648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 293.311524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 297.449843] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:44:06 (1776919446) [ 297.544541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.579264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.701578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 297.837986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 301.672468] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:44:10 (1776919450) [ 301.745324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.771848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.865683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 301.979640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 306.015267] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:44:14 (1776919454) [ 306.084665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.109734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.396407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 306.588784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 310.214585] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:44:18 (1776919458) [ 310.284552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.300267] LNet: 20426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 310.306377] LNet: 20426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 310.322877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.371237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 310.402974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 313.898751] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:44:22 (1776919462) [ 313.966643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.000905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.059816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 314.085479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 317.909689] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:44:26 (1776919466) [ 318.021402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.050380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.107583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 322.003995] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:44:30 (1776919470) [ 322.114168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.153478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.196199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 325.549682] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:44:34 (1776919474) [ 325.609654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.639976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.689349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 325.889693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 328.991370] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:44:37 (1776919477) [ 329.034348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.056657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.105768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 329.121891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 332.064103] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:44:40 (1776919480) [ 332.131871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.156568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.193930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 332.215245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 335.091406] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:44:43 (1776919483) [ 335.175434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.195720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.237916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 335.256758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 338.268129] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:44:47 (1776919487) [ 338.359472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.389828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.448586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 338.479871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 341.994534] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:44:50 (1776919490) [ 342.078213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.118821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.158642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 342.207883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 342.221735] LNetError: 25740:0:(peer.c:2174:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 342.248959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 342.272404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 345.542281] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:44:54 (1776919494) [ 345.598341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.631609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.689454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.f38rE2BlMz [ 345.739792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.f38rE2BlMz [ 345.783600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.f38rE2BlMz [ 349.104798] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:44:57 (1776919497) [ 349.186772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.215449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.251521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 349.278797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 349.308425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 352.265205] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:45:00 (1776919500) [ 352.344602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.362653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.397367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 353.641546] LNet: Message forwarding will be enabled in 2s [ 356.704220] LNet: Message forwarding enabled [ 356.866671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 356.876985] LNet: Message forwarding will stop in 2s to 6s [ 361.112467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 361.616355] LNet: Message forwarding disabled [ 362.136875] LNet: Message forwarding will be enabled in 2s [ 365.186658] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:45:13 (1776919513) [ 365.274115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.880635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 365.916989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 368.554378] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:45:17 (1776919517) [ 368.601320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.622354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.660635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 368.677147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 368.694971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 368.712939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 368.735530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 368.751907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 368.771639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 368.799334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 368.819977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 368.838770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 368.859284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 368.882349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 368.906591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 368.925417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 368.953591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 368.973508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 368.997373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 369.021229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 369.044813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 369.060934] LNetError: 28808:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 372.223672] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:45:20 (1776919520) [ 372.270978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.301433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.345386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.514946] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:45:24 (1776919524) [ 376.356566] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg305-server [ 377.203643] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:45:25 (1776919525) [ 377.252253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.267114] LNet: 30303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.270212] LNet: 30303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 15 previous similar messages [ 377.284487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.317845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 377.329835] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 378.196468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[6-10]@tcp [ 378.208272] LNet: 30523:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 378.237623] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 378.254090] LNetError: 30313:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.6@tcp has gone from up to down [ 379.233910] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 379.244502] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 380.260754] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 380.269529] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 381.556268] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:45:30 (1776919530) [ 382.305704] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.6@tcp is being used as a gateway but routing feature is not turned on [ 382.322206] LNetError: 30313:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 382.568445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg305-server [ 383.461851] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:45:32 (1776919532) [ 383.557468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.569456] LNetError: 31458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 383.583139] LNet: Removed LNI 192.168.203.5@tcp [ 383.599653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.633214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 383.653403] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 383.999817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[6-11/2]@tcp [ 384.010630] LNet: 31575:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 384.016987] LNet: 31575:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 384.030535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[6-11/2]@tcp [ 384.031090] LNetError: 31462:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.8@tcp has gone from up to down [ 384.048044] LNetError: 31462:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 387.276793] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:45:35 (1776919535) [ 387.336171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.350516] LNetError: 32162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 387.371241] LNet: Removed LNI 192.168.203.5@tcp [ 387.387411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.436292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 387.462926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.485946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.518644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 387.555246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.583164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.619832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 387.665570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.688305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.725658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 387.770790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.794760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.837638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 387.874313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.899126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.934357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 387.972931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.994775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.023916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 388.062118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.090143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.129575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 388.173966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.198398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.232399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 388.280205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.301292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.344801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 388.405093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.426617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.462396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 388.503706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.534111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.582548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 392.049319] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:45:40 (1776919540) [ 392.987270] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg305-server [ 393.965672] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:45:42 (1776919542) [ 394.884495] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg305-server [ 395.826759] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:45:44 (1776919544) [ 396.258854] Key type .llcrypt unregistered [ 396.260830] Key type ._llcrypt unregistered [ 399.294723] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 399.416794] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 399.423505] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 399.430902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 401.698332] Key type ._llcrypt registered [ 401.700590] Key type .llcrypt registered [ 401.813542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 401.822528] alg: No test for adler32 (adler32-zlib) [ 402.660705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.691328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 402.706989] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 403.108415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 403.123921] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 404.247897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 404.259667] LNet: Removed LNI 10.1.2.3@tcp [ 404.330772] LNet: 34286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.335788] LNetError: 34286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 404.350248] LNet: Removed LNI 192.168.203.5@tcp [ 404.672555] Key type .llcrypt unregistered [ 404.675214] Key type ._llcrypt unregistered [ 409.889760] Key type ._llcrypt registered [ 409.891312] Key type .llcrypt registered [ 410.584699] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:45:59 (1776919559) [ 410.920390] Key type .llcrypt unregistered [ 410.928092] Key type ._llcrypt unregistered [ 413.823391] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 413.883056] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 413.887744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 413.890808] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 415.163563] Key type ._llcrypt registered [ 415.169539] Key type .llcrypt registered [ 415.255965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 415.263807] alg: No test for adler32 (adler32-zlib) [ 416.112768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.157135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 416.176676] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 416.566489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 416.582051] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 417.901164] LNet: 35769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.906772] LNetError: 35769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 417.912867] LNet: Removed LNI 192.168.203.5@tcp [ 417.919343] LNet: Removed LNI 10.1.2.3@tcp [ 418.277630] Key type .llcrypt unregistered [ 418.279943] Key type ._llcrypt unregistered [ 423.979614] Key type ._llcrypt registered [ 423.982161] Key type .llcrypt registered [ 424.706598] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:46:13 (1776919573) [ 425.009341] Key type .llcrypt unregistered [ 425.011554] Key type ._llcrypt unregistered [ 428.309745] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 428.399679] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 428.407057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 428.410895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 430.723783] Key type ._llcrypt registered [ 430.725657] Key type .llcrypt registered [ 430.832676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 430.843411] alg: No test for adler32 (adler32-zlib) [ 431.703438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.739263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 431.763332] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 432.791789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 432.807834] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 433.922867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 433.938267] LNet: Removed LNI 10.1.2.3@tcp [ 433.949626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 433.974915] LNet: Removed LNI 10.1.2.31@tcp [ 433.987781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 434.007680] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 434.386582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 434.398247] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 434.906368] LNet: 37704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.915596] LNetError: 37704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 434.920594] LNet: Removed LNI 10.1.2.3@tcp [ 434.930448] LNet: Removed LNI 10.1.2.31@tcp [ 435.244525] Key type .llcrypt unregistered [ 435.247325] Key type ._llcrypt unregistered [ 440.003460] Key type ._llcrypt registered [ 440.004919] Key type .llcrypt registered [ 440.652464] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:46:29 (1776919589) [ 440.838269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 440.847883] alg: No test for adler32 (adler32-zlib) [ 441.691163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.724735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 441.736794] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 442.159380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.174773] LNet: 38655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.178853] LNetError: 38655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 442.192688] LNet: Removed LNI 192.168.203.5@tcp [ 442.207989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.239393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.5@tcp [ 442.261080] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 442.619552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.5@tcp [ 442.644326] LNet: Removed LNI 192.168.203.5@tcp [ 442.676520] LNet: 38790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.999681] Key type .llcrypt unregistered [ 443.005069] Key type ._llcrypt unregistered [ 448.208151] Key type ._llcrypt registered [ 448.209975] Key type .llcrypt registered [ 448.889239] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:46:37 (1776919597) [ 449.851080] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg305-server [ 450.873266] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:46:39 (1776919599) [ 451.171534] Key type .llcrypt unregistered [ 451.173852] Key type ._llcrypt unregistered [ 454.631809] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 454.756376] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 454.760435] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 454.764444] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 457.098658] Key type ._llcrypt registered [ 457.101285] Key type .llcrypt registered [ 457.210924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 457.220371] alg: No test for adler32 (adler32-zlib) [ 458.049272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.099547] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 458.488426] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 459.712885] LNet: 40638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 459.717880] LNetError: 40638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 459.722493] LNet: Removed LNI 192.168.203.5@tcp1 [ 459.728675] LNet: Removed LNI 10.1.2.3@tcp2 [ 460.092456] Key type .llcrypt unregistered [ 460.097047] Key type ._llcrypt unregistered [ 465.849640] Key type ._llcrypt registered [ 465.851935] Key type .llcrypt registered [ 466.508996] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:46:55 (1776919615) [ 466.817364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 466.831347] alg: No test for adler32 (adler32-zlib) [ 467.668347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.707955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 467.733083] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 468.259921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.277301] LNet: 41604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.288395] LNetError: 41604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 468.304542] LNet: Removed LNI 192.168.203.5@tcp [ 468.326282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.373327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 468.391683] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 471.918756] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:47:00 (1776919620) [ 472.069606] LNet: 42194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.075791] LNetError: 42194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 472.086304] LNet: Removed LNI 192.168.203.5@tcp [ 472.410658] Key type .llcrypt unregistered [ 472.412697] Key type ._llcrypt unregistered [ 475.272231] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 475.342525] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 475.345387] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 475.348589] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 477.668690] Key type ._llcrypt registered [ 477.670936] Key type .llcrypt registered [ 477.778937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 477.790380] alg: No test for adler32 (adler32-zlib) [ 478.627775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.656659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 478.674485] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 479.073556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 479.085895] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 480.105031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.114820] LNet: 42936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 480.117798] LNetError: 42936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 480.121787] LNet: Removed LNI 192.168.203.5@tcp [ 480.128481] LNet: Removed LNI 10.1.2.3@tcp [ 480.139306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.169686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 480.186740] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 480.189972] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 480.412471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.432252] LNet: Removed LNI 192.168.203.5@tcp [ 480.441306] LNet: Removed LNI 10.1.2.3@tcp [ 480.453776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.492647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 480.514726] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 480.519216] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 480.734264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 480.761535] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 481.376332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.390319] LNet: 43126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.396539] LNet: 43126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 481.401414] LNetError: 43126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 481.406442] LNetError: 43126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 481.414676] LNet: Removed LNI 192.168.203.5@tcp [ 481.417913] LNet: Removed LNI 10.1.2.3@tcp [ 481.431586] LNet: Removed LNI 192.168.203.5@tcp2 [ 481.446447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.481675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 481.499849] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 481.502801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 481.506363] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 484.694911] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:47:13 (1776919633) [ 484.754088] LNet: 43714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 484.763492] LNetError: 43714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 484.770735] LNetError: 43714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 484.776675] LNet: Removed LNI 192.168.203.5@tcp [ 484.779562] LNet: Removed LNI 10.1.2.3@tcp [ 484.789021] LNet: Removed LNI 192.168.203.5@tcp2 [ 485.132924] Key type .llcrypt unregistered [ 485.135702] Key type ._llcrypt unregistered [ 487.986590] Key type ._llcrypt registered [ 487.987872] Key type .llcrypt registered [ 488.085768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 488.104224] alg: No test for adler32 (adler32-zlib) [ 488.962381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.996799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 489.013966] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 489.474635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.486720] LNet: 44116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 489.492792] LNetError: 44116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 489.508586] LNet: Removed LNI 192.168.203.5@tcp [ 489.521427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.555632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 489.572113] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 489.785765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.810416] LNet: Removed LNI 192.168.203.5@tcp [ 489.823386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.868917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 489.897976] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 490.137101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.152776] LNet: 44176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 490.155971] LNet: 44176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 490.161125] LNetError: 44176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 490.168234] LNetError: 44176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 490.176834] LNet: Removed LNI 192.168.203.5@tcp [ 490.187750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.232911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 490.249919] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 490.478893] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 490.547780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 490.549660] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 490.553765] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 492.729206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.741355] LNet: 44253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.746644] LNetError: 44253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 492.762192] LNet: Removed LNI 192.168.203.5@tcp [ 492.777374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.812646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 492.829182] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 495.862272] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:47:24 (1776919644) [ 495.989446] LNet: 44849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 495.994485] LNetError: 44849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 496.015546] LNet: Removed LNI 192.168.203.5@tcp [ 496.322748] Key type .llcrypt unregistered [ 496.325106] Key type ._llcrypt unregistered [ 499.201626] Key type ._llcrypt registered [ 499.203586] Key type .llcrypt registered [ 499.324570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 499.334478] alg: No test for adler32 (adler32-zlib) [ 500.156576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.185804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 500.199676] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 500.717173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.730782] LNet: 45247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.735753] LNetError: 45247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.750603] LNet: Removed LNI 192.168.203.5@tcp [ 500.763321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.806835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 500.822677] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 504.198437] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:47:32 (1776919652) [ 504.247120] LNet: 45833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 504.256513] LNetError: 45833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 504.267407] LNet: Removed LNI 192.168.203.5@tcp [ 504.598240] Key type .llcrypt unregistered [ 504.602329] Key type ._llcrypt unregistered [ 507.365218] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 507.429645] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 507.434136] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 507.437556] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 509.698823] Key type ._llcrypt registered [ 509.700965] Key type .llcrypt registered [ 509.782956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 509.792241] alg: No test for adler32 (adler32-zlib) [ 510.624840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.656294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 510.672523] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 511.060828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.070103] LNet: 46289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 511.073610] LNetError: 46289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 511.081855] LNet: Removed LNI 192.168.203.5@tcp [ 511.095119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.132585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 511.153553] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 514.385468] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:47:43 (1776919663) [ 514.435860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.460565] LNet: 46880:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 514.464541] LNetError: 46880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 514.476799] LNet: Removed LNI 192.168.203.5@tcp [ 514.494210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.566958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 517.706290] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:47:46 (1776919666) [ 517.772325] LNet: 47462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.140571] Key type .llcrypt unregistered [ 518.143426] Key type ._llcrypt unregistered [ 520.646361] Key type ._llcrypt registered [ 520.647430] Key type .llcrypt registered [ 520.747181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 520.756376] alg: No test for adler32 (adler32-zlib) [ 521.578503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.614547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 521.630727] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 522.068979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.086971] LNet: 47864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 522.091726] LNetError: 47864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 522.101357] LNet: Removed LNI 192.168.203.5@tcp [ 522.112788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.146158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 522.160056] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 525.634305] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:47:54 (1776919674) [ 525.676641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.693944] LNet: 48449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 525.701539] LNetError: 48449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 525.713091] LNet: Removed LNI 192.168.203.5@tcp [ 525.726350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.768310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 528.785810] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:47:57 (1776919677) [ 529.521354] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 530.362081] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:47:59 (1776919679) [ 531.079149] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 531.789069] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:48:00 (1776919680) [ 532.503895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 533.241562] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:48:02 (1776919682) [ 533.899141] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 534.696633] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:48:03 (1776919683) [ 535.469460] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 536.219064] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:48:04 (1776919684) [ 536.256682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.270294] LNet: 50500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.285500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.319239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 536.330566] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 536.696986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.5 [ 536.708678] LNet: 50640:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 536.713457] LNet: Removed LNI 192.168.203.5@tcp [ 539.448575] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:48:08 (1776919688) [ 539.522754] LNet: 51209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 539.525302] LNetError: 51209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 539.535812] LNet: Removed LNI 192.168.203.5@tcp [ 539.807613] Key type .llcrypt unregistered [ 539.809545] Key type ._llcrypt unregistered [ 542.237177] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 542.398926] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 542.402778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 542.406919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 544.795310] Key type ._llcrypt registered [ 544.799240] Key type .llcrypt registered [ 544.876385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 544.887118] alg: No test for adler32 (adler32-zlib) [ 545.918855] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 549.001629] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:48:17 (1776919697) [ 549.053473] LNet: 52106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.056989] LNetError: 52106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 549.067182] LNet: Removed LNI 10.1.2.3@tcp [ 549.374728] Key type .llcrypt unregistered [ 549.376734] Key type ._llcrypt unregistered [ 551.969571] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 552.110143] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 552.113149] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 552.120656] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 554.452316] Key type ._llcrypt registered [ 554.454014] Key type .llcrypt registered [ 554.527101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 554.536720] alg: No test for adler32 (adler32-zlib) [ 555.381897] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 558.311687] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:48:27 (1776919707) [ 558.381982] LNet: 53005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.389119] LNetError: 53005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 558.398072] LNet: Removed LNI 10.1.2.3@tcp [ 558.709668] Key type .llcrypt unregistered [ 558.711434] Key type ._llcrypt unregistered [ 561.095036] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 561.209275] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 561.211819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 561.214471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 563.582050] Key type ._llcrypt registered [ 563.583451] Key type .llcrypt registered [ 563.661530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 563.670309] alg: No test for adler32 (adler32-zlib) [ 564.517081] LNet: 53306:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 564.524480] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 567.355667] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:48:36 (1776919716) [ 567.400745] LNet: 53889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 567.404588] LNetError: 53889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 567.412908] LNet: Removed LNI 10.1.2.3@tcp [ 567.718453] Key type .llcrypt unregistered [ 567.720289] Key type ._llcrypt unregistered [ 570.308841] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 570.444710] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 570.447047] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 570.452695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 572.831043] Key type ._llcrypt registered [ 572.832662] Key type .llcrypt registered [ 572.930344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 572.940544] alg: No test for adler32 (adler32-zlib) [ 573.773228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.804737] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 577.112827] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:48:45 (1776919725) [ 577.332755] LNet: 55125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 577.336321] LNetError: 55125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 577.349618] LNet: Removed LNI 10.1.2.3@tcp [ 577.636720] Key type .llcrypt unregistered [ 577.642171] Key type ._llcrypt unregistered [ 580.168467] Key type ._llcrypt registered [ 580.170365] Key type .llcrypt registered [ 580.258818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 580.266057] alg: No test for adler32 (adler32-zlib) [ 581.099067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 581.121477] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 584.867485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 585.227840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 585.251398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 585.273226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 585.284682] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 585.387610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 585.406106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 585.423432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 585.440539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 585.459705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 585.480203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 585.498162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 585.520546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 585.531161] LNet: 55561:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 585.544940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 585.566071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 585.588394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 585.611279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 585.627592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 585.650498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 585.675776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 585.938792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 585.959407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 585.981431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 585.993907] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 586.090690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 586.112647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 586.134610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 586.158669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 586.177443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 586.196361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 586.216470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 586.237123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 586.248147] LNet: 55670:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 586.261209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 586.271448] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 586.283501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 586.299824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 586.319412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 586.338388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 586.354665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 586.373838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 586.683334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 586.702034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 586.718702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 586.837535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 586.858117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 586.873343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 586.890588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 586.916805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 586.939675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 586.962412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 586.985523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 587.016731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 587.035927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 587.058663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 587.085725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 587.110505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 587.133091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 587.151350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 587.420828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 587.443496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 587.467862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 587.480167] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 587.483102] LNet: Skipped 2 previous similar messages [ 587.582425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 587.598982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 587.614643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 587.631359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 587.653916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 587.670547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 587.690503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 587.708875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 587.722772] LNet: 55888:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 587.726809] LNet: 55888:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 587.736978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 587.754602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 587.772664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 587.791712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 587.810708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 587.829442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 587.851978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 588.100545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 588.124467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 588.147761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 588.250519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 588.266678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 588.282394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 588.298863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 588.322320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 588.339667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 588.369671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 588.391971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 588.411296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 588.428939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 588.447702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 588.462921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 588.481110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 588.503513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 588.526860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 588.766650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 588.790968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 588.821728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 588.927872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 588.942426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 588.961929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 588.982294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 589.001717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 589.022626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 589.040298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 589.058186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 589.076735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 589.099924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 589.119891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 589.137379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 589.154636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 589.171473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 589.189351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 589.302867] LNet: 56148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.307662] LNetError: 56148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.323663] LNet: Removed LNI 192.168.203.5@tcp [ 589.508193] Key type .llcrypt unregistered [ 589.510274] Key type ._llcrypt unregistered [ 597.210928] Key type ._llcrypt registered [ 597.212670] Key type .llcrypt registered [ 597.889413] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:49:06 (1776919746) [ 598.269634] Key type .llcrypt unregistered [ 598.272460] Key type ._llcrypt unregistered [ 600.967908] Key type ._llcrypt registered [ 600.969586] Key type .llcrypt registered [ 601.125087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 601.134238] alg: No test for adler32 (adler32-zlib) [ 601.974119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 601.991914] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 606.242357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 611.227760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.253808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 611.270716] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 611.956363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 611.978945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 1 [ 612.009485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 612.038559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 1 [ 612.071112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_interrupt -r 1 [ 612.094837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 1 [ 612.119901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_interrupt -r 1 [ 612.147694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 1 [ 612.170070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 612.181407] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 612.184679] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 612.285391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 612.302830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 612.323996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 612.341300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 612.362364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 612.382494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 612.404207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 612.423156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 612.439663] LNet: 57610:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 612.449693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 612.465267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 612.493483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 612.516909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 612.538543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 612.558445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 612.579180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 612.821422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 612.846356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 1 [ 612.870413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 612.895506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 1 [ 612.927897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_dropped -r 1 [ 612.951121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 1 [ 612.970435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_dropped -r 1 [ 612.987113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 1 [ 613.010357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 613.020271] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 613.023795] LNet: Skipped 2 previous similar messages [ 613.140898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 613.157243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 613.179899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 613.196785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 613.217195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 613.235059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 613.252539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 613.270757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 613.283218] LNet: 57731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 613.296606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 613.318919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 613.346315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 613.370616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 613.389753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 613.411338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 613.435395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 613.680776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 613.701596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 1 [ 613.719955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 613.743157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 1 [ 613.770260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_aborted -r 1 [ 613.792732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 1 [ 613.813395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_aborted -r 1 [ 613.832259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 1 [ 613.850425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 613.969275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 613.992727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 614.016518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 614.043757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 614.067601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 614.085870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 614.109850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 614.133036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 614.158752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 614.181247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 614.203649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 614.228764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 614.256451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 614.287451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 614.316493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 614.566530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 614.588926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 1 [ 614.609935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 614.630240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 1 [ 614.657098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_no_route -r 1 [ 614.684954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 1 [ 614.706600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_no_route -r 1 [ 614.727965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 1 [ 614.753880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 614.768731] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 614.772432] LNet: Skipped 6 previous similar messages [ 614.872815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 614.893093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 614.910306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 614.927083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 614.942706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 614.957769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 614.974410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 614.991982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 615.005363] LNet: 57973:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 615.008936] LNet: 57973:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 615.017300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 615.033671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 615.048943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 615.071617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 615.094201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 615.114735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 615.137360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 615.384726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 615.414416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 1 [ 615.439675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 615.461719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 1 [ 615.482904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_timeout -r 1 [ 615.508152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 1 [ 615.529362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_timeout -r 1 [ 615.551304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 1 [ 615.570470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 615.714779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 615.738233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 615.762632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 615.782700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 615.802786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 615.819859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 615.843989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 615.862561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 615.883306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 615.902035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 615.918530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 615.935947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 615.954644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 615.972480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 615.995570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 616.237492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 616.255372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_error -r 1 [ 616.274610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 616.292896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_error -r 1 [ 616.313316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_error -r 1 [ 616.333155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_error -r 1 [ 616.353782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_error -r 1 [ 616.374761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_error -r 1 [ 616.397402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 616.504719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 616.520326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 616.536546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 616.554616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 616.572942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 616.592357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 616.608096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 616.628376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 616.647815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 616.665156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 616.681905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 616.699045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 616.717808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 616.736942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 616.752187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 619.280786] LNet: 58291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.283799] LNetError: 58291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.288271] LNet: Removed LNI 192.168.203.5@tcp [ 620.328169] LNet: Removed LNI 192.168.203.5@tcp1 [ 620.532525] Key type .llcrypt unregistered [ 620.534302] Key type ._llcrypt unregistered [ 627.501622] Key type ._llcrypt registered [ 627.504309] Key type .llcrypt registered [ 628.180798] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:49:36 (1776919776) [ 628.586730] Key type .llcrypt unregistered [ 628.590566] Key type ._llcrypt unregistered [ 631.142672] Key type ._llcrypt registered [ 631.144692] Key type .llcrypt registered [ 631.253962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 631.264516] alg: No test for adler32 (adler32-zlib) [ 632.097299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 632.125866] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 635.939700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 636.234933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 636.256549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 636.274506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 636.286881] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 636.390476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 636.408678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 636.427597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 636.447495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 636.465858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 636.485340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 636.504437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 636.523537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 636.534490] LNet: 59536:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 636.546726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 636.564829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 636.580619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 636.600147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 636.619236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 636.639062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 636.656189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 636.941482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 636.975314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 637.007212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 637.019508] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 637.133244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 637.151992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 637.170357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 637.188578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 637.205877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 637.227433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 637.249450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 637.268772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 637.280466] LNet: 59645:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 637.290820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 637.307492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 637.324100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 637.340887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 637.369081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 637.387834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 637.409140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 637.669428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 637.695800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 637.732899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 637.878497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 637.900409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 637.924189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 637.958362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 637.984996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 638.015193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 638.032645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 638.056254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 638.073413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 638.093562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 638.114164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 638.128437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 638.147431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 638.169588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 638.190741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 638.329876] LNet: 59795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.334198] LNetError: 59795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 638.347937] LNet: Removed LNI 192.168.203.5@tcp [ 638.551530] Key type .llcrypt unregistered [ 638.553794] Key type ._llcrypt unregistered [ 645.612743] Key type ._llcrypt registered [ 645.615037] Key type .llcrypt registered [ 646.277179] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:49:54 (1776919794) [ 646.712354] Key type .llcrypt unregistered [ 646.714171] Key type ._llcrypt unregistered [ 649.245134] Key type ._llcrypt registered [ 649.246931] Key type .llcrypt registered [ 649.339505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 649.347153] alg: No test for adler32 (adler32-zlib) [ 650.195579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 650.213029] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 653.995321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 657.600663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.626509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 657.646249] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 658.433326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 658.450969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 1 [ 658.471341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 658.493801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 1 [ 658.528778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_dropped -r 1 [ 658.554698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 1 [ 658.582654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_dropped -r 1 [ 658.609788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 1 [ 658.637699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 658.653498] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 658.660283] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 658.783700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 658.806641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 658.831402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 658.851535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 658.876406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 658.893616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 658.916660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 658.939596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 658.955433] LNet: 61258:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 658.967875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 658.991957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 659.011724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 659.035592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 659.057153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 659.080216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 659.107243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 659.133155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 659.421849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 659.449242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_error -r 1 [ 659.469845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 659.488646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_error -r 1 [ 659.516389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_error -r 1 [ 659.552139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_error -r 1 [ 659.578219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_error -r 1 [ 659.605505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_error -r 1 [ 659.635223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 659.649573] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 659.653594] LNet: Skipped 2 previous similar messages [ 659.772691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 659.794200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 659.814653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 659.839297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 659.864944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 659.899718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 659.921301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 659.940090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 659.955947] LNet: 61382:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 659.970679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 659.993825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 660.018799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 660.043436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 660.061739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 660.089592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 660.109733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 660.137806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 660.464287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 660.482880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_timeout -r 1 [ 660.504317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 660.533800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_timeout -r 1 [ 660.560239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_timeout -r 1 [ 660.590632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_timeout -r 1 [ 660.617974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_timeout -r 1 [ 660.646498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_timeout -r 1 [ 660.670430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.105@tcp [ 660.685397] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 660.688967] LNet: Skipped 1 previous similar message [ 660.811455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 660.827297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 660.843110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 660.860588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 660.885611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 660.909502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 660.931629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 660.955162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 660.968283] LNet: 61505:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 660.979952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 661.002207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 661.021079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 661.038168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 661.057603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 661.079807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 661.104845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 661.133232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 663.870514] LNet: 61582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.873687] LNetError: 61582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.877755] LNet: Removed LNI 192.168.203.5@tcp [ 663.892326] LNet: Removed LNI 192.168.203.5@tcp1 [ 664.085044] Key type .llcrypt unregistered [ 664.088394] Key type ._llcrypt unregistered [ 671.471248] Key type ._llcrypt registered [ 671.472424] Key type .llcrypt registered [ 672.255434] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:50:20 (1776919820) [ 672.551348] Key type .llcrypt unregistered [ 672.555934] Key type ._llcrypt unregistered [ 675.554149] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 675.653980] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 675.656266] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 675.660550] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 677.969842] Key type ._llcrypt registered [ 677.972145] Key type .llcrypt registered [ 678.044838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 678.053365] alg: No test for adler32 (adler32-zlib) [ 678.900465] LNet: 62655:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 678.909202] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 678.928939] LNet: 62669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.933146] LNetError: 62669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.947603] LNet: Removed LNI 192.168.203.5@tcp [ 679.296492] Key type .llcrypt unregistered [ 679.298206] Key type ._llcrypt unregistered [ 681.948980] Key type ._llcrypt registered [ 681.951163] Key type .llcrypt registered [ 682.048611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 682.056879] alg: No test for adler32 (adler32-zlib) [ 682.888784] LNet: 62927:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 682.902938] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 682.906413] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 682.930526] LNet: 62941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.935389] LNetError: 62941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.939494] LNet: Removed LNI 192.168.203.5@tcp [ 682.948812] LNet: Removed LNI 10.1.2.3@tcp1 [ 683.274433] Key type .llcrypt unregistered [ 683.276332] Key type ._llcrypt unregistered [ 685.729787] Key type ._llcrypt registered [ 685.733816] Key type .llcrypt registered [ 685.818898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 685.826893] alg: No test for adler32 (adler32-zlib) [ 686.645558] LNet: 63200:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 686.659290] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 686.661201] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 686.680869] LNet: 63214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.687348] LNetError: 63214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.693215] LNet: Removed LNI 192.168.203.5@tcp [ 686.701520] LNet: Removed LNI 10.1.2.3@tcp [ 686.980791] Key type .llcrypt unregistered [ 686.982969] Key type ._llcrypt unregistered [ 689.597664] Key type ._llcrypt registered [ 689.599202] Key type .llcrypt registered [ 689.681432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 689.689109] alg: No test for adler32 (adler32-zlib) [ 690.511700] LNet: 63473:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 690.520134] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 690.523116] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 690.540110] LNet: 63487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.544323] LNetError: 63487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.547693] LNet: Removed LNI 192.168.203.5@tcp [ 690.557843] LNet: Removed LNI 10.1.2.3@tcp [ 690.845360] Key type .llcrypt unregistered [ 690.846938] Key type ._llcrypt unregistered [ 693.158307] Key type ._llcrypt registered [ 693.159677] Key type .llcrypt registered [ 693.229568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 693.236911] alg: No test for adler32 (adler32-zlib) [ 694.054296] LNet: 63751:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 694.060996] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 694.063085] LNetError: 63751:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 694.066550] LNetError: Error -100 starting up LNI tcp [ 694.075381] LNet: Removed LNI 192.168.203.5@tcp [ 694.358400] Key type .llcrypt unregistered [ 694.360125] Key type ._llcrypt unregistered [ 698.847721] Key type ._llcrypt registered [ 698.849180] Key type .llcrypt registered [ 699.374823] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:50:48 (1776919848) [ 699.703512] Key type .llcrypt unregistered [ 699.706031] Key type ._llcrypt unregistered [ 702.252589] Key type ._llcrypt registered [ 702.255790] Key type .llcrypt registered [ 702.370260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 702.380431] alg: No test for adler32 (adler32-zlib) [ 703.203596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 703.216580] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 706.641663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 706.901373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 706.918657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 706.937323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 706.947798] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 707.032866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 707.046907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 707.060901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 707.076216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 707.092682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 707.111125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 707.127346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 707.147926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 707.160527] LNet: 65007:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 707.171602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 707.187186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 707.202299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 707.217233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 707.233499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 707.250156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 707.359455] LNet: 65046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.362808] LNetError: 65046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.374993] LNet: Removed LNI 192.168.203.5@tcp [ 707.542440] Key type .llcrypt unregistered [ 707.544322] Key type ._llcrypt unregistered [ 713.992639] Key type ._llcrypt registered [ 713.994625] Key type .llcrypt registered [ 714.075070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.082805] alg: No test for adler32 (adler32-zlib) [ 714.902163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 714.915438] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 718.130881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 721.251605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.266875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 721.279926] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 721.859242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 721.878598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e network_timeout -r 1 [ 721.897166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 721.920377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e network_timeout -r 1 [ 721.940699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e network_timeout -r 1 [ 721.959473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e network_timeout -r 1 [ 721.978439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e network_timeout -r 1 [ 721.996903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e network_timeout -r 1 [ 722.015165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 722.024694] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 722.079233] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 722.082492] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 722.121205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 722.137077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 722.157508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 722.176908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 722.195360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 722.211760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 722.228780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 722.249625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 722.259694] LNet: 65921:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 722.269706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 722.287250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 722.302491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 722.319515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 722.334462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 722.353526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 724.688715] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 724.851930] LNet: 65994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.856316] LNetError: 65994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 724.861985] LNet: Removed LNI 192.168.203.5@tcp [ 724.877434] LNet: Removed LNI 192.168.203.5@tcp1 [ 725.088476] Key type .llcrypt unregistered [ 725.090036] Key type ._llcrypt unregistered [ 731.812870] Key type ._llcrypt registered [ 731.814798] Key type .llcrypt registered [ 732.409946] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 00:51:21 (1776919881) [ 732.582736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 732.590440] alg: No test for adler32 (adler32-zlib) [ 733.438295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.468819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.482111] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 733.838897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 733.850657] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 734.252263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 734.288684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 734.311515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 734.334691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 734.352703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 734.371385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 734.389387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 734.400170] LNetError: 67092:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 734.495425] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 734.495458] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 736.543234] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 737.639369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 740.642045] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 740.896088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 740.917130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 740.946976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.958712] LNet: 67199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.963667] LNetError: 67199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.968769] LNet: Removed LNI 192.168.203.5@tcp [ 740.980422] LNet: Removed LNI 192.168.203.5@tcp1 [ 740.991809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.019043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 741.031304] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 741.430337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 741.441903] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 741.829304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 741.850258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 741.880339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 741.901978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 741.920208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 741.939903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 741.959409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 741.981929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 741.994621] LNetError: 67471:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 744.095681] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 744.098746] LNet: Skipped 1 previous similar message [ 744.154679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 748.191341] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 752.604318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 752.623667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 752.643855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 755.713786] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 00:51:44 (1776919904) [ 755.756022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.767413] LNet: 68182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.770841] LNetError: 68182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.774053] LNetError: 68182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 755.778473] LNet: Removed LNI 192.168.203.5@tcp [ 755.786149] LNet: Removed LNI 192.168.203.5@tcp1 [ 755.798298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.826668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.838553] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 756.184096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 756.194345] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 756.560335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 756.588629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 756.605216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 756.622418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 756.642265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 756.660087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 756.677327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 0 [ 756.722857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 756.831139] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.5@tcp [ 757.855374] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 757.858975] LNet: Skipped 1 previous similar message [ 771.285531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 771.357348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 771.376144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 771.397154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 500 [ 771.452892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 777.745323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 777.776972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.788106] LNet: 68678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.791947] LNetError: 68678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.796507] LNetError: 68678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 777.801715] LNet: Removed LNI 192.168.203.5@tcp [ 777.807997] LNet: Removed LNI 192.168.203.5@tcp1 [ 777.823342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.856898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 777.871203] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 778.336798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 778.347776] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 778.781892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 778.802844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 778.838477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 778.860661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 778.885574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 778.906634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 778.938173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 778.966351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.5@tcp --health 0 [ 779.871335] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 779.878862] LNet: Skipped 3 previous similar messages [ 780.134160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 790.710496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 790.732273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 790.748945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 793.688727] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 00:52:22 (1776919942) [ 793.892762] LNet: 69734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.896644] LNetError: 69734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.901082] LNetError: 69734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 793.905658] LNet: Removed LNI 192.168.203.5@tcp [ 793.915655] LNet: Removed LNI 192.168.203.5@tcp1 [ 794.226310] Key type .llcrypt unregistered [ 794.227931] Key type ._llcrypt unregistered [ 796.587541] Key type ._llcrypt registered [ 796.590424] Key type .llcrypt registered [ 796.674387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 796.681370] alg: No test for adler32 (adler32-zlib) [ 797.507355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 797.526735] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 801.673536] Lustre: DEBUG MARKER: Initial discovery [ 801.742086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 802.975632] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 803.053595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 803.076388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 803.100890] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 803.125863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 803.162351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.105@tcp -r 1 -m GET -l 3 [ 803.186297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.105@tcp -r 1 -m GET -e local_error [ 803.210356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.105@tcp -r 1 -m PUT -l 6 [ 803.679165] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 803.943520] Lustre: DEBUG MARKER: Force oleg305-client.virtnet to discover 192.168.203.105@tcp (in background) [ 803.984597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.105@tcp [ 804.758820] Lustre: DEBUG MARKER: Force oleg305-server to discover 192.168.203.5@tcp [ 805.897162] Lustre: DEBUG MARKER: Wait for 70487 [ 807.007284] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 807.763831] Lustre: DEBUG MARKER: Finished wait on 70487 [ 807.849763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 807.877741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 808.011344] LNet: 70944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.021625] LNetError: 70944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.038943] LNet: Removed LNI 192.168.203.5@tcp [ 808.257976] Key type .llcrypt unregistered [ 808.262523] Key type ._llcrypt unregistered [ 815.081318] Key type ._llcrypt registered [ 815.083414] Key type .llcrypt registered [ 815.799584] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 00:52:44 (1776919964) [ 816.073472] Key type .llcrypt unregistered [ 816.075178] Key type ._llcrypt unregistered [ 819.299756] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 819.400884] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 819.403667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 819.406318] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 821.694735] Key type ._llcrypt registered [ 821.696990] Key type .llcrypt registered [ 821.800129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 821.809397] alg: No test for adler32 (adler32-zlib) [ 822.684450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 822.719577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 822.732527] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 823.142965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 823.155447] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 827.996502] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 00:52:56 (1776919976) [ 828.059965] LNet: 73009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 828.064243] LNetError: 73009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 828.072073] LNet: Removed LNI 192.168.203.5@tcp [ 828.079544] LNet: Removed LNI 10.1.2.3@tcp [ 828.446500] Key type .llcrypt unregistered [ 828.449915] Key type ._llcrypt unregistered [ 831.737834] Key type ._llcrypt registered [ 831.739713] Key type .llcrypt registered [ 831.879894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 831.892785] alg: No test for adler32 (adler32-zlib) [ 832.738625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.779385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 832.794973] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 833.256253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 833.270176] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 833.966314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.5@tcp [ 835.040153] LNetError: Unexpected error -22 connecting to 192.168.203.5@tcp at host 192.168.203.5:988 [ 837.916325] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 00:53:06 (1776919986) [ 837.962853] LNet: 74141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.969590] LNetError: 74141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.975656] LNet: Removed LNI 192.168.203.5@tcp [ 837.989027] LNet: Removed LNI 10.1.2.3@tcp [ 838.389138] Key type .llcrypt unregistered [ 838.390801] Key type ._llcrypt unregistered [ 841.709179] Key type ._llcrypt registered [ 841.710990] Key type .llcrypt registered [ 841.830333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 841.839807] alg: No test for adler32 (adler32-zlib) [ 842.710333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.747813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 842.777966] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 843.200984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 843.214844] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 843.763858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.5@tcp1 --nid 192.168.203.5@tcp2 [ 846.040746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp1 192.168.203.5@tcp2 [ 846.059416] LNetError: 74983:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.5@tcp1 to 192.168.203.5@tcp2 - Route table may be misconfigured [ 846.083304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp2 192.168.203.5@tcp1 [ 849.013091] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 00:53:17 (1776919997) [ 849.048327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.059869] LNet: 75550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.063050] LNetError: 75550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.066137] LNet: Removed LNI 192.168.203.5@tcp1 [ 850.089240] LNet: Removed LNI 192.168.203.5@tcp2 [ 850.111904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.149399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 850.163356] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 850.505801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 850.517408] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 850.960773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 850.981940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.5@tcp [ 851.003717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp1 192.168.203.5@tcp1 [ 851.030905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp -d 192.168.203.5@tcp -e network_timeout [ 851.056424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -e network_timeout [ 851.078364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -e network_timeout [ 851.099914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -e network_timeout [ 851.122093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 851.133404] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 851.143655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 851.167489] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 851.194340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 851.245436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 854.481089] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 00:53:23 (1776920003) [ 854.563152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.579875] LNet: 76413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.582816] LNetError: 76413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.586350] LNetError: 76413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 854.590988] LNet: Removed LNI 192.168.203.5@tcp [ 855.655682] LNet: Removed LNI 192.168.203.5@tcp1 [ 855.681960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.739691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 855.893782] LNet: 76454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.091300] Key type .llcrypt unregistered [ 856.092792] Key type ._llcrypt unregistered [ 861.314892] Key type ._llcrypt registered [ 861.317476] Key type .llcrypt registered [ 861.942209] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 00:53:30 (1776920010) [ 862.155626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 862.166155] alg: No test for adler32 (adler32-zlib) [ 863.002839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.724173] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 864.550248] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 00:53:33 (1776920013) [ 864.603959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.620804] LNet: 77548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.635560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.661742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 864.673783] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 865.040088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 865.050658] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 865.499298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 865.523693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp1 [ 865.546199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp1 [ 868.623493] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 00:53:37 (1776920017) [ 869.497997] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg305-server [ 870.262593] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 00:53:39 (1776920019) [ 871.018360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg305-server [ 871.796604] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 00:53:40 (1776920020) [ 872.612642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg305-server [ 873.305582] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 00:53:42 (1776920022) [ 874.055216] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg305-server [ 874.773133] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 00:53:43 (1776920023) [ 875.623490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg305-server [ 876.358698] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 00:53:45 (1776920025) [ 877.219708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg305-server [ 877.990767] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 00:53:46 (1776920026) [ 878.867649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg305-server [ 879.729924] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 00:53:48 (1776920028) [ 880.667755] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg305-server [ 881.406285] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 00:53:50 (1776920030) [ 882.376277] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg305-server [ 883.233294] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 00:53:51 (1776920031) [ 883.287359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.301841] LNet: 81419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.305753] LNetError: 81419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.311935] LNet: Removed LNI 192.168.203.5@tcp [ 884.325649] LNet: Removed LNI 192.168.203.5@tcp1 [ 884.339631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.367300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.380501] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 884.744593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 884.779224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 884.877491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 884.896983] LNet: 81570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 884.903351] LNetError: 81570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 884.908461] LNetError: 81570:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 884.921462] LNet: Removed LNI 192.168.203.5@tcp [ 884.939167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.973169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 884.993091] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 885.402988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 885.490297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.565497] LNet: Removed LNI 192.168.203.5@tcp [ 886.578293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.614546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 886.631185] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 887.039272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 887.084719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 887.206804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 887.221293] LNet: 81862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 887.224523] LNet: 81862:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 887.228367] LNetError: 81862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 887.236691] LNetError: 81862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 887.255553] LNet: Removed LNI 192.168.203.5@tcp [ 887.272130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.301139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 887.321931] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 887.754080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 887.851275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.934528] LNet: Removed LNI 192.168.203.5@tcp [ 888.952253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.990449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 889.004857] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 889.360992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 889.403106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 889.514563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.529864] LNetError: 82155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.534448] LNetError: 82155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 889.547511] LNet: Removed LNI 192.168.203.5@tcp [ 889.558155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.585873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 889.606434] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 890.037500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 890.164374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 891.243394] LNet: Removed LNI 192.168.203.5@tcp [ 891.262108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.298838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 891.314777] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 891.672891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 891.715395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 892.859095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.874262] LNet: 82456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.877505] LNet: 82456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 893.925653] LNet: Removed LNI 192.168.203.5@tcp [ 893.937571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.970966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 893.984899] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 894.343867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 894.462411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.476312] LNetError: 82602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.480275] LNetError: 82602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 894.491898] LNet: Removed LNI 192.168.203.5@tcp [ 894.506192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.538731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.554895] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 894.933223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 894.976634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 897.243963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.280197] LNet: Removed LNI 192.168.203.5@tcp [ 898.294598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.324844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 898.344230] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 898.750197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 898.890786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.943291] LNet: Removed LNI 192.168.203.5@tcp [ 899.957235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.992792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 900.011972] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 900.401436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 900.442238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 901.646070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 901.661827] LNet: 83069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.667406] LNet: 83069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 902.694082] LNet: Removed LNI 192.168.203.5@tcp [ 902.706854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.737728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 902.754919] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 903.182982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 903.349650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 903.367939] LNetError: 83214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.371692] LNetError: 83214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 904.421751] LNet: Removed LNI 192.168.203.5@tcp [ 904.434161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.471087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 904.490200] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 904.915651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 904.956672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.979881] LNet: Removed LNI 192.168.203.5@tcp [ 904.993216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.030576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 905.054888] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 905.494493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 909.233720] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 00:54:17 (1776920057) [ 909.308338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.333265] LNet: Removed LNI 192.168.203.5@tcp [ 909.348463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.378443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 909.394347] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 910.036138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.061575] LNet: Removed LNI 192.168.203.5@tcp231 [ 910.076766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.154523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 910.171920] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 910.178929] LNet: 84199:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 910.427642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 910.455445] LNet: Removed LNI 192.168.203.5@tcp231 [ 910.471317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 910.493685] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 911.177825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.213652] LNet: Removed LNI 192.168.203.5@tcp231 [ 911.233666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.278583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 911.304842] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/0] [ 911.308711] LNet: 84351:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 911.543693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 911.568252] LNet: Removed LNI 192.168.203.5@tcp231 [ 911.582692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 911.598596] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/0] [ 912.299707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.321332] LNet: Removed LNI 192.168.203.5@tcp231 [ 912.335349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.377793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 912.395717] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/60] [ 912.406835] LNet: 84502:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 912.657206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 912.680030] LNet: Removed LNI 192.168.203.5@tcp231 [ 912.693054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 912.714903] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/60] [ 913.316795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.336158] LNet: Removed LNI 192.168.203.5@tcp231 [ 913.353782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.393902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 913.407841] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 913.616550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 913.643633] LNet: Removed LNI 192.168.203.5@tcp231 [ 913.656251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 913.676967] LNet: Added LNI 192.168.203.5@tcp231 [8/256/0/180] [ 917.758099] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 00:54:26 (1776920066) [ 917.836673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.856187] LNet: 85363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.861598] LNet: 85363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 917.885173] LNet: Removed LNI 192.168.203.5@tcp231 [ 917.904685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.949149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 917.963401] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 918.341155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 918.369278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 918.397183] LNet: Removed LNI 192.168.203.5@tcp232 [ 918.409423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 918.432065] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 918.764076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 918.782784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 918.805589] LNet: Removed LNI 192.168.203.5@tcp232 [ 918.819618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 918.841693] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 919.186455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 919.208682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 919.228160] LNet: Removed LNI 192.168.203.5@tcp232 [ 919.238212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 919.253211] LNet: Added LNI 192.168.203.5@tcp232 [8/256/0/180] [ 919.622178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 919.639942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 919.656212] LNet: Removed LNI 192.168.203.5@tcp232 [ 919.666776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 922.406187] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 00:54:31 (1776920071) [ 922.859213] Key type .llcrypt unregistered [ 922.861456] Key type ._llcrypt unregistered [ 925.071304] Key type ._llcrypt registered [ 925.072870] Key type .llcrypt registered [ 925.159317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 925.169928] alg: No test for adler32 (adler32-zlib) [ 925.993515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 926.009941] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 929.414311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 933.179279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.198433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 933.209901] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 933.805513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_interrupt -r 2 [ 933.823726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 2 [ 933.842259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_interrupt -r 2 [ 933.860876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 2 [ 933.883083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_interrupt -r 2 [ 933.903169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_interrupt -r 2 [ 933.921477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_interrupt -r 2 [ 933.944581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_interrupt -r 2 [ 933.966899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 933.981516] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 933.984762] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 934.087067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 934.101814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 934.117689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 934.133219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 934.151401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 934.168350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 934.184700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 934.200801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 934.212398] LNet: 87150:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 934.223985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 934.239584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 934.257088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 934.272992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 934.288451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 934.304862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 934.539731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_dropped -r 2 [ 934.560951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 2 [ 934.580492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_dropped -r 2 [ 934.600794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 2 [ 934.619436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_dropped -r 2 [ 934.640186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_dropped -r 2 [ 934.665789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_dropped -r 2 [ 934.686412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_dropped -r 2 [ 934.704091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 934.715171] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 934.814414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 934.830367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 934.845839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 934.862461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 934.878447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 934.893447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 934.911837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 934.929757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 934.939766] LNet: 87270:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 934.949304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 934.965689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 934.985357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 935.000472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 935.018703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 935.038661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 935.274749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_aborted -r 2 [ 935.292640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 2 [ 935.311569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_aborted -r 2 [ 935.336230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 2 [ 935.359225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_aborted -r 2 [ 935.379508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_aborted -r 2 [ 935.400815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_aborted -r 2 [ 935.424827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_aborted -r 2 [ 935.446684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 935.557626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 935.574871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 935.596871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 935.613746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 935.632669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 935.648342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 935.665921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 935.686907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 935.706491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 935.722131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 935.742137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 935.759423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 935.779563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 935.801470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 936.073358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_no_route -r 2 [ 936.095167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 2 [ 936.117138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_no_route -r 2 [ 936.138717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 2 [ 936.161471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_no_route -r 2 [ 936.186868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_no_route -r 2 [ 936.204124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_no_route -r 2 [ 936.226464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_no_route -r 2 [ 936.247785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 936.261749] LNet: There was an unexpected network error while writing to 192.168.203.105: rc = -22 [ 936.265331] LNet: Skipped 2 previous similar messages [ 936.374414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 936.390438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 936.406637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 936.423775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 936.453366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 936.473568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 936.492691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 936.509607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 936.518441] LNet: 87510:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 936.522238] LNet: 87510:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 936.534450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 936.560360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 936.577435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 936.595644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 936.614085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 936.633199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 936.905139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e local_timeout -r 2 [ 936.930502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 2 [ 936.952391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e local_timeout -r 2 [ 936.978836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 2 [ 937.001413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e local_timeout -r 2 [ 937.030158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e local_timeout -r 2 [ 937.061642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e local_timeout -r 2 [ 937.091532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e local_timeout -r 2 [ 937.120699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 937.252940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 937.269995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 937.293694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 937.314682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 937.337776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 937.364054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 937.387417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 937.407936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 937.424867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 937.443501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 937.463250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 937.478300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 937.494340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 937.511319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 937.736661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp -m GET -e remote_dropped -r 2 [ 937.756618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 2 [ 937.776729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp -m GET -e remote_dropped -r 2 [ 937.797945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.5@tcp -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 2 [ 937.816889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp -m GET -e remote_dropped -r 2 [ 937.836183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.105@tcp1 -m GET -e remote_dropped -r 2 [ 937.859934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp -m GET -e remote_dropped -r 2 [ 937.878672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.5@tcp1 -d 192.168.203.5@tcp1 -m GET -e remote_dropped -r 2 [ 937.896406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.5@tcp 192.168.203.105@tcp [ 937.906977] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 938.012741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 938.030974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 938.048820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 938.068602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 938.088244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 938.109641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 938.126151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 938.142575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 938.160712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 938.177633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 938.196625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 938.218418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 938.240466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 938.264324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 940.941625] LNet: 87824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.949746] LNetError: 87824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.955887] LNet: Removed LNI 192.168.203.5@tcp [ 940.963499] LNet: Removed LNI 192.168.203.5@tcp1 [ 941.164798] Key type .llcrypt unregistered [ 941.166746] Key type ._llcrypt unregistered [ 947.294766] Key type ._llcrypt registered [ 947.296395] Key type .llcrypt registered [ 947.929384] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 00:54:56 (1776920096) [ 948.118610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.125735] alg: No test for adler32 (adler32-zlib) [ 948.960303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.990954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 949.006879] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 949.393143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 949.413071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.5@tcp [ 949.425543] LNetError: 88800:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 949.485160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.5@tcp [ 949.531903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 952.430795] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 00:55:01 (1776920101) [ 952.508339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.528993] LNet: 89387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.532435] LNetError: 89387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.541976] LNet: Removed LNI 192.168.203.5@tcp [ 952.553773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.592667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 952.608736] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 952.970461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 952.981638] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 953.493398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.5@tcp [ 953.505597] LNetError: 89654:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.5@tcp to recovery [ 953.632584] LNet: 1 local NIs in recovery (showing 1): 192.168.203.5@tcp [ 953.634339] LNet: There was an unexpected network error while writing to 192.168.203.5: rc = -22 [ 954.625833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.5@tcp [ 958.628536] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 00:55:07 (1776920107) [ 959.507808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg305-server [ 960.381907] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 00:55:09 (1776920109) [ 961.226487] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg305-server [ 961.953085] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 00:55:10 (1776920110) [ 962.731561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg305-server [ 963.440112] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 00:55:12 (1776920112) [ 963.501088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.513471] LNet: 91274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.517875] LNetError: 91274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.521532] LNet: Removed LNI 192.168.203.5@tcp [ 963.528859] LNet: Removed LNI 192.168.203.5@tcp1 [ 963.538568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.569591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 963.584250] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 963.907535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 963.933993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 963.954780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 963.987683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 964.007772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 964.028381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 966.642903] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 00:55:15 (1776920115) [ 966.675583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.687082] LNet: 91994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.693115] LNetError: 91994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.696375] LNetError: 91994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 966.708329] LNet: Removed LNI 192.168.203.5@tcp [ 966.717495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.746878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 966.757693] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 969.857901] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 00:55:18 (1776920118) [ 969.898492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.911185] LNet: 92696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.914526] LNetError: 92696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.924633] LNet: Removed LNI 192.168.203.5@tcp [ 969.941272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.968802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 969.983727] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 973.040842] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 00:55:21 (1776920121) [ 973.678453] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 974.473525] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 00:55:23 (1776920123) [ 974.694118] LNet: 93736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.697875] LNetError: 93736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.708573] LNet: Removed LNI 192.168.203.5@tcp [ 975.038784] Key type .llcrypt unregistered [ 975.041313] Key type ._llcrypt unregistered [ 977.692199] Key type ._llcrypt registered [ 977.694029] Key type .llcrypt registered [ 977.783228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 977.791715] alg: No test for adler32 (adler32-zlib) [ 978.605694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 978.622668] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 981.974273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 983.797957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.105@tcp [ 983.960556] LNet: 94152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.966314] LNetError: 94152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.986399] LNet: Removed LNI 192.168.203.5@tcp [ 984.168371] Key type .llcrypt unregistered [ 984.169760] Key type ._llcrypt unregistered [ 988.773452] Key type ._llcrypt registered [ 988.775515] Key type .llcrypt registered [ 989.431723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 990.173982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 990.972506] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 00:55:39 (1776920139) [ 991.127312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 991.137550] alg: No test for adler32 (adler32-zlib) [ 991.968688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.008519] LNet: 95237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.293251] Key type .llcrypt unregistered [ 992.294973] Key type ._llcrypt unregistered [ 995.085831] Key type ._llcrypt registered [ 995.087631] Key type .llcrypt registered [ 995.183762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 995.196219] alg: No test for adler32 (adler32-zlib) [ 996.029253] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 996.031910] LNet: 95493:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 996.040047] LNet: 95493:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 996.056093] LNetError: 95505:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.8@tcp is being used as a gateway but routing feature is not turned on [ 996.072160] LNetError: 95505:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.8@tcp has gone from up to down [ 996.296438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1015.775165] LNetError: 95505:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.6@tcp has gone from up to down [ 1015.782037] LNetError: 95505:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 1015.788513] LNet: 95551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.796505] LNetError: 95551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.822049] LNet: Removed LNI 192.168.203.5@tcp [ 1015.844606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1015.886898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1015.912769] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1015.930946] LNet: 95561:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1015.953086] LNet: 95561:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 1015.966124] LNetError: 95561:0:(peer.c:366:lnet_peer_ni_del_locked()) Peer NI 192.168.203.7@tcp is a gateway. Can not delete it [ 1015.977374] LNetError: 95561:0:(peer.c:2166:lnet_del_peer_ni()) 192.168.203.7@tcp is a router. Can not be deleted [ 1015.986219] LNet: 95561:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.051178] LNetError: 95558:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.8@tcp is being used as a gateway but routing feature is not turned on [ 1016.075235] LNetError: 95558:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 1019.039659] LNetError: 95565:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) route to o2ib through 192.168.203.6@tcp has gone from up to down [ 1019.080914] LNetError: 95565:0:(lib-lnet.h:1336:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 1024.189614] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 00:56:12 (1776920172) [ 1026.060474] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg305-server [ 1027.780883] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 00:56:15 (1776920175) [ 1029.783882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg305-server [ 1031.928518] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 00:56:19 (1776920179) [ 1034.027717] Lustre: Lustre: Build Version: 2.17.51_74_g68f2299 [ 1035.969777] Key type lgssc registered [ 1037.649058] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1059.834507] Key type lgssc unregistered [ 1060.657217] LNet: 97982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.673622] LNetError: 97982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.703624] LNet: Removed LNI 192.168.203.5@tcp [ 1061.569914] Key type .llcrypt unregistered [ 1061.571559] Key type ._llcrypt unregistered [ 1076.663546] Key type ._llcrypt registered [ 1076.665178] Key type .llcrypt registered [ 1077.975313] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 00:57:05 (1776920225) [ 1078.492676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1078.510872] alg: No test for adler32 (adler32-zlib) [ 1079.396960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1079.443097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1079.504559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1079.566302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1079.616575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1084.424299] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 00:57:12 (1776920232) [ 1084.544064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1084.568161] LNet: 99386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.589852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.642143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1084.658431] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1085.472237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.497963] LNet: 99533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.505426] LNetError: 99533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.519178] LNet: Removed LNI 192.168.203.5@tcp [ 1085.554371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.623503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1085.651814] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1086.165454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1086.193642] LNetError: 99561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.218454] LNet: Removed LNI 192.168.203.5@tcp [ 1086.240495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1086.311896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1086.348704] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1091.606414] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 00:57:19 (1776920239) [ 1091.698137] LNet: 100144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.701201] LNet: 100144:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1091.706949] LNetError: 100144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.731929] LNet: Removed LNI 192.168.203.5@tcp [ 1092.408105] Key type .llcrypt unregistered [ 1092.412323] Key type ._llcrypt unregistered [ 1097.551294] Key type ._llcrypt registered [ 1097.553972] Key type .llcrypt registered [ 1097.720462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1097.735725] alg: No test for adler32 (adler32-zlib) [ 1098.662179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.739731] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1098.743203] LNet: *** cfs_fail_loc=e003, val=100*** [ 1099.350555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1099.369775] LNet: *** cfs_fail_loc=e003, val=100*** [ 1099.372440] LNet: Skipped 2 previous similar messages [ 1099.382624] Ping reply received with mapping: [ 1099.386357] found metadata reply NI[192.168.203.5@tcp] [ 1099.389311] 05 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1099.391989] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1099.394475] 5a 5a 5a 5a 5a 5a 5a 5a [ 1099.493421] LNet: 100549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.501420] LNetError: 100549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.519622] LNet: Removed LNI 192.168.203.5@tcp [ 1101.112498] Key type .llcrypt unregistered [ 1101.116078] Key type ._llcrypt unregistered [ 1108.961486] Key type ._llcrypt registered [ 1108.963480] Key type .llcrypt registered [ 1109.810552] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 00:57:38 (1776920258) [ 1110.180281] Key type .llcrypt unregistered [ 1110.184116] Key type ._llcrypt unregistered [ 1113.732137] Key type ._llcrypt registered [ 1113.733532] Key type .llcrypt registered [ 1113.916522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1113.931871] alg: No test for adler32 (adler32-zlib) [ 1114.944928] LNetError: 101594:0:(api-ni.c:2731:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1115.204827] Key type .llcrypt unregistered [ 1115.207431] Key type ._llcrypt unregistered [ 1119.023346] Key type ._llcrypt registered [ 1119.026455] Key type .llcrypt registered [ 1120.051871] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 00:57:48 (1776920268) [ 1120.447418] Key type .llcrypt unregistered [ 1120.450781] Key type ._llcrypt unregistered [ 1124.416871] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1124.421190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1126.848059] Key type ._llcrypt registered [ 1126.852805] Key type .llcrypt registered [ 1126.997185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1127.011533] alg: No test for adler32 (adler32-zlib) [ 1127.861658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1127.907337] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1134.830325] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 00:58:03 (1776920283) [ 1134.936837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1134.958307] LNet: 103370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1134.971575] LNetError: 103370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1134.987203] LNet: Removed LNI 10.1.2.3@tcp [ 1135.009905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.583776] Key type .llcrypt unregistered [ 1135.585926] Key type ._llcrypt unregistered [ 1139.927943] Key type ._llcrypt registered [ 1139.929624] Key type .llcrypt registered [ 1140.061515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1140.077298] alg: No test for adler32 (adler32-zlib) [ 1140.953881] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1141.013327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1141.140874] LNet: 103661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.147313] LNetError: 103661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1141.162807] LNet: Removed LNI 192.168.203.5@tcp [ 1141.595413] Key type .llcrypt unregistered [ 1141.597535] Key type ._llcrypt unregistered [ 1148.836687] Key type ._llcrypt registered [ 1148.838748] Key type .llcrypt registered [ 1149.764605] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 00:58:18 (1776920298) [ 1150.304310] Key type .llcrypt unregistered [ 1150.308720] Key type ._llcrypt unregistered [ 1155.409742] Key type ._llcrypt registered [ 1155.413654] Key type .llcrypt registered [ 1155.604146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1155.623368] alg: No test for adler32 (adler32-zlib) [ 1161.735618] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 00:58:29 (1776920309) [ 1161.886336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1161.935356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1162.024330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1162.060815] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1162.651298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1162.685237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1162.722957] LNet: Removed LNI 192.168.203.5@tcp [ 1162.968993] LNet: 105404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.388433] Key type .llcrypt unregistered [ 1163.391459] Key type ._llcrypt unregistered [ 1172.568997] Key type ._llcrypt registered [ 1172.571165] Key type .llcrypt registered [ 1173.734058] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 00:58:41 (1776920321) [ 1174.048982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1174.065968] alg: No test for adler32 (adler32-zlib) [ 1174.940995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1175.006356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1175.041824] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1175.788543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1175.820071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.5@tcp [ 1175.844505] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1175.851956] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1175.855437] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1175, 0, 0 [ 1175.862452] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1175.870455] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1175.874754] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1175.883071] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1175, 0, 0 [ 1175.886813] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1175.893023] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1175.898166] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1175.902166] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1175, 0, 1225 [ 1175.907390] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1175.909946] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1175.913191] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1175.918272] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1175, 0, 1225 [ 1175.922440] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1181.387156] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 00:58:49 (1776920329) [ 1181.752545] LNet: 106946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.761926] LNetError: 106946:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1181.771981] LNet: Removed LNI 192.168.203.5@tcp [ 1182.252530] Key type .llcrypt unregistered [ 1182.254469] Key type ._llcrypt unregistered [ 1187.003179] Key type ._llcrypt registered [ 1187.011991] Key type .llcrypt registered [ 1187.179439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1187.193550] alg: No test for adler32 (adler32-zlib) [ 1188.130870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1188.158049] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1194.210628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1200.736913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1200.771660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1200.802467] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1201.998319] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1202.185181] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1202.190100] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1202.194696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1204.515183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1204.541469] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1212.104558] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 00:59:20 (1776920360) [ 1212.159977] LNet: 108443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.167413] LNetError: 108443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.172595] LNet: Removed LNI 192.168.203.5@tcp [ 1213.215437] LNet: Removed LNI 192.168.203.5@tcp1 [ 1213.224402] LNetError: 108443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1213.238948] LNetError: 108443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1213.261220] LNet: Removed LNI 10.1.2.3@tcp99 [ 1213.692144] Key type .llcrypt unregistered [ 1213.694716] Key type ._llcrypt unregistered [ 1216.539497] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1216.542316] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1218.760594] Key type ._llcrypt registered [ 1218.762059] Key type .llcrypt registered [ 1218.838646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1218.844552] alg: No test for adler32 (adler32-zlib) [ 1219.663693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1219.691468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1219.703743] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1220.042607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1220.056652] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1221.059935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1221.084526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1221.112802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1221.128457] LNetError: 109195:0:(peer.c:2174:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1221.151316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1221.174055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1221.356599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1221.388092] LNet: 109215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.393265] LNetError: 109215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.396317] LNet: Removed LNI 192.168.203.5@tcp [ 1221.402545] LNet: Removed LNI 10.1.2.3@tcp [ 1221.654712] Key type .llcrypt unregistered [ 1221.656208] Key type ._llcrypt unregistered [ 1226.240536] Key type ._llcrypt registered [ 1226.242223] Key type .llcrypt registered [ 1226.883572] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 00:59:35 (1776920375) [ 1227.066639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1227.075396] alg: No test for adler32 (adler32-zlib) [ 1227.895287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.934479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1227.947201] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1228.308571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.5@tcp [ 1228.352576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg305-client@tcp [ 1231.328510] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 00:59:40 (1776920380) [ 1231.386432] LNet: 110742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.391510] LNetError: 110742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1231.403179] LNet: Removed LNI 192.168.203.5@tcp [ 1231.686524] Key type .llcrypt unregistered [ 1231.688593] Key type ._llcrypt unregistered [ 1234.101879] Key type ._llcrypt registered [ 1234.103037] Key type .llcrypt registered [ 1234.170595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1234.178089] alg: No test for adler32 (adler32-zlib) [ 1235.000172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1235.012195] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1235.028289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1237.754655] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 00:59:46 (1776920386) [ 1238.397110] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1239.097564] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 00:59:47 (1776920387) [ 1239.713971] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1240.391256] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 00:59:49 (1776920389) [ 1240.446484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1240.458527] LNet: 112171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.462616] LNetError: 112171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.477152] LNet: Removed LNI 192.168.203.5@tcp [ 1240.489804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1240.516689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1240.534768] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1240.854415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1240.878366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1243.653703] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 00:59:52 (1776920392) [ 1243.694322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1243.705149] LNet: 112875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.707850] LNetError: 112875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.720184] LNet: Removed LNI 192.168.203.5@tcp [ 1243.733168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.765082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1243.782586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1244.103257] Key type .llcrypt unregistered [ 1244.104979] Key type ._llcrypt unregistered [ 1248.760173] Key type ._llcrypt registered [ 1248.761850] Key type .llcrypt registered [ 1249.335927] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 00:59:58 (1776920398) [ 1249.507624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1249.519960] alg: No test for adler32 (adler32-zlib) [ 1250.339780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1250.365613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1250.378799] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1250.711216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1250.740772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1253.877223] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:00:02 (1776920402) [ 1253.926373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1253.941944] LNet: 114430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.945388] LNetError: 114430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.960451] LNet: Removed LNI 192.168.203.5@tcp [ 1253.973996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1254.007655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1254.027649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1256.955224] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:00:05 (1776920405) [ 1257.003487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1257.016063] LNet: 115013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.028682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1257.068116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1257.244738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1257.261812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1257.299978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1257.480778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1257.505619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1257.544224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1257.570335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1257.611905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1257.790820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1257.817385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1257.854480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1258.039193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1258.058695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1260.870120] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:00:09 (1776920409) [ 1260.944426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1260.957685] LNet: 115649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.960773] LNet: 115649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1260.974140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1261.006034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1261.018780] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1261.400794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1261.412561] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1261.817353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1261.854879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1261.979459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1261.993206] LNetError: 115928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.000056] LNet: Removed LNI 192.168.203.5@tcp [ 1262.009437] LNet: Removed LNI 192.168.203.5@tcp2 [ 1262.023364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1262.071120] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1262.075736] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1262.079203] LNet: 115934:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1262.328719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1262.434529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1263.455136] LNet: Removed LNI 192.168.203.5@tcp [ 1263.456746] LNetError: 115971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.460676] LNetError: 115971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1263.472298] LNet: Removed LNI 192.168.203.5@tcp2 [ 1263.486950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1263.526214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1263.541320] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1263.890905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1263.903655] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1264.353353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1264.375766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1264.408256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1264.509429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1264.522640] LNet: 116255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.526345] LNet: 116255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1265.567090] LNet: Removed LNI 192.168.203.5@tcp [ 1265.573455] LNetError: 116255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1265.577302] LNetError: 116255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1265.586666] LNet: Removed LNI 192.168.203.5@tcp2 [ 1265.598378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1265.635427] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1265.640336] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1265.644418] LNet: 116261:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1265.862244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1265.966649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1265.984843] LNet: Removed LNI 192.168.203.5@tcp [ 1267.047710] LNet: Removed LNI 192.168.203.5@tcp2 [ 1267.068267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1267.094367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1267.105989] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1267.451962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1267.466355] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1267.913159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1267.965182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1268.077851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1269.151143] LNet: Removed LNI 192.168.203.5@tcp [ 1269.159758] LNet: Removed LNI 192.168.203.5@tcp2 [ 1269.178389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.220804] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1269.224307] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1269.227897] LNet: 116584:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.454800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1269.553629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1269.566191] LNet: 116621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.571916] LNet: 116621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1269.577407] LNetError: 116621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.582404] LNetError: 116621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1269.589380] LNet: Removed LNI 192.168.203.5@tcp [ 1269.600455] LNet: Removed LNI 192.168.203.5@tcp2 [ 1269.612118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.651654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1269.672600] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1270.012689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1270.028792] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1270.441357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1270.471053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1270.502900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1270.607688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1271.647314] LNet: Removed LNI 192.168.203.5@tcp [ 1272.677481] LNet: Removed LNI 192.168.203.5@tcp2 [ 1272.692664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.724266] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1272.727634] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1272.730961] LNet: 116911:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1272.955466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1273.074626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1273.086748] LNet: Removed LNI 192.168.203.5@tcp [ 1274.149874] LNet: Removed LNI 192.168.203.5@tcp2 [ 1274.163460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1274.191950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1274.205599] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1274.543913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1274.556974] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1274.966766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1274.982748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1275.009890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1275.182848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1276.255713] LNet: Removed LNI 192.168.203.5@tcp [ 1276.266930] LNet: Removed LNI 192.168.203.5@tcp2 [ 1276.281196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1276.327416] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1276.331267] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1276.551958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1276.714440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1277.727179] LNet: Removed LNI 192.168.203.5@tcp [ 1277.728452] LNetError: 117303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.735161] LNetError: 117303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1277.744494] LNet: Removed LNI 192.168.203.5@tcp2 [ 1277.757456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1277.783518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1277.800422] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1278.143889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1278.153890] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1278.518019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1278.542066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1278.558257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1278.587336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1278.744403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1278.756415] LNet: 117605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.760098] LNet: 117605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1279.775129] LNet: Removed LNI 192.168.203.5@tcp [ 1280.808238] LNet: Removed LNI 192.168.203.5@tcp2 [ 1280.829995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.883899] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1280.889444] LNet: Added LNI 192.168.203.5@tcp2 [8/256/0/180] [ 1280.893471] LNet: 117611:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.899634] LNet: 117611:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1281.146057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1284.249711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1285.024260] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:00:33 (1776920433) [ 1285.137111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1285.150915] LNet: Removed LNI 192.168.203.5@tcp [ 1286.182417] LNet: Removed LNI 192.168.203.5@tcp2 [ 1286.201129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1286.343785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1286.361588] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1286.695921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1286.709981] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1287.177359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1287.191162] LNet: Removed LNI 192.168.203.5@tcp [ 1287.205565] LNet: Removed LNI 10.1.2.3@tcp [ 1287.221913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.261537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1287.277436] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1287.280903] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1287.481934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1287.494592] LNet: Removed LNI 192.168.203.5@tcp [ 1287.501882] LNet: Removed LNI 10.1.2.3@tcp [ 1290.608859] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:00:39 (1776920439) [ 1290.665698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1290.688853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1290.746021] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1290.828873] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1290.831367] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1290.834274] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1292.981557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1292.995622] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1293.386963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1293.403473] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1294.318271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.5@tcp [ 1294.338504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1294.372499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.5@tcp -m PUT -e local_timeout -r 1 [ 1294.404219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1294.434867] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1294.439133] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1294.550292] LNetError: 119309:0:(lib-move.c:4969:lnet_parse()) 192.168.203.5@tcp, src 192.168.203.5@tcp: Dropping PUT (error -108 looking up sender) [ 1295.584479] LNet: 119767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.587275] LNet: 119767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1295.591472] LNetError: 119767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.594675] LNetError: 119767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1295.597884] LNet: Removed LNI 192.168.203.5@tcp [ 1295.605474] LNet: Removed LNI 10.1.2.3@tcp [ 1295.801675] Key type .llcrypt unregistered [ 1295.803785] Key type ._llcrypt unregistered [ 1300.288863] Key type ._llcrypt registered [ 1300.290122] Key type .llcrypt registered [ 1300.883810] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:00:49 (1776920449) [ 1301.039942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1301.051666] alg: No test for adler32 (adler32-zlib) [ 1301.875261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.141459] LNet: 120618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1302.502764] Key type .llcrypt unregistered [ 1302.504898] Key type ._llcrypt unregistered [ 1305.177455] Key type ._llcrypt registered [ 1305.180512] Key type .llcrypt registered [ 1305.264897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1305.272675] alg: No test for adler32 (adler32-zlib) [ 1306.111991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1306.127401] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1309.848661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1309.933666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1310.192322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.105@tcp [ 1310.241440] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 1310.313298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1310.326936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1310.342817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1310.357550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1310.373809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1310.394421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1310.416082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1310.438347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1310.450120] LNet: 121082:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1310.463582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1310.481681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1310.507401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1310.533177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1310.553935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1310.577515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1310.936197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1311.076082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1311.095379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1311.117706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1311.138772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1311.156575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1311.176306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1311.204611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1311.224744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1311.236577] LNet: 121199:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1311.251983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1311.275725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1311.302474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1311.326260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1311.351952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1311.378210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1311.765827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.105@tcp [ 1311.911962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1311.945714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1311.968814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1311.988561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1312.009755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1312.032797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1312.053798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1312.071392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1312.089568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1312.106471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1312.121985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1312.139558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1312.155697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1312.172254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1312.514522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1312.652386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1312.671961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1312.697781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1312.720667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1312.741871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1312.761832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1312.785807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1312.807500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1312.817573] LNet: 121446:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1312.819612] LNet: 121446:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1312.831692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1312.852872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1312.872173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1312.889228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1312.913484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1312.931927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1313.107442] LNet: 121492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.113152] LNetError: 121492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1314.150316] LNet: Removed LNI 192.168.203.5@tcp [ 1314.362613] Key type .llcrypt unregistered [ 1314.364075] Key type ._llcrypt unregistered [ 1321.698806] Key type ._llcrypt registered [ 1321.701529] Key type .llcrypt registered [ 1322.427706] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:01:10 (1776920470) [ 1322.730508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1322.739873] alg: No test for adler32 (adler32-zlib) [ 1323.584187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1323.833986] LNet: 122378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.199518] Key type .llcrypt unregistered [ 1324.201499] Key type ._llcrypt unregistered [ 1326.912528] Key type ._llcrypt registered [ 1326.915993] Key type .llcrypt registered [ 1327.039780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1327.053052] alg: No test for adler32 (adler32-zlib) [ 1327.891914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1327.908194] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1331.597397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.105@tcp [ 1335.288396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.304469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1335.316224] LNet: Added LNI 192.168.203.5@tcp1 [8/256/0/180] [ 1335.743976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1336.035342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.105@tcp [ 1336.095107] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.105@tcp [ 1336.160558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1336.175666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1336.190927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1336.205422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1336.220693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1336.236685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1336.257479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1336.277842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1336.287375] LNet: 123029:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1336.298420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1336.317736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1336.336383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1336.357947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1336.379729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.396843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1336.707802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1336.821049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1336.840543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1336.857841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1336.871992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1336.887686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1336.901758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1336.922526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1336.945375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1336.961233] LNet: 123146:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1336.976518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1337.004508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1337.031084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1337.056259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1337.079726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1337.101920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1337.445750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.105@tcp [ 1337.553454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1337.575631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1337.595435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1337.615240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1337.637384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1337.652770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1337.670527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1337.689772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1337.707538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1337.724744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1337.743159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1337.767803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1337.788201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1337.806342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1338.124739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1338.247351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1338.263700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1338.287069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.303903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.322452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1338.338443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1338.354736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1338.370799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1338.382859] LNet: 123380:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1338.386932] LNet: 123380:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1338.397566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1338.413467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1338.431169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1338.448804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1338.467901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1338.485953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.339687] LNet: 123461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.343270] LNetError: 123461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.347579] LNet: Removed LNI 192.168.203.5@tcp [ 1341.353472] LNet: Removed LNI 192.168.203.5@tcp1 [ 1341.521971] Key type .llcrypt unregistered [ 1341.526066] Key type ._llcrypt unregistered [ 1348.099122] Key type ._llcrypt registered [ 1348.100364] Key type .llcrypt registered [ 1348.650202] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:01:37 (1776920497) [ 1351.568822] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:01:40 (1776920500) [ 1355.095454] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:01:43 (1776920503) [ 1358.699370] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:01:47 (1776920507) [ 1362.148444] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:01:50 (1776920510) [ 1366.269865] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:01:54 (1776920514) [ 1369.868806] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:01:58 (1776920518) [ 1373.451524] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:02:02 (1776920522) [ 1377.102449] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:02:05 (1776920525) [ 1380.580022] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:02:09 (1776920529) [ 1384.366943] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:02:13 (1776920533) [ 1387.763258] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:02:16 (1776920536) [ 1391.145076] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:02:19 (1776920539) [ 1394.448436] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:02:23 (1776920543) [ 1398.545383] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:02:27 (1776920547) [ 1402.519200] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:02:31 (1776920551) [ 1405.619694] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:02:34 (1776920554) [ 1408.788598] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:02:37 (1776920557) [ 1408.929405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1408.936741] alg: No test for adler32 (adler32-zlib) [ 1409.760687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1409.826976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1409.844508] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1410.392407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1410.402132] LNet: 134161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.406698] LNetError: 134161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.423586] LNet: Removed LNI 192.168.203.5@tcp [ 1410.433628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.458696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1410.481288] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1413.666636] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:02:42 (1776920562) [ 1417.044080] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:02:45 (1776920565) [ 1420.377353] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:02:49 (1776920569) [ 1424.009558] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:02:52 (1776920572) [ 1427.765720] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:02:56 (1776920576) [ 1431.294259] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:03:00 (1776920580) [ 1434.687090] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:03:03 (1776920583) [ 1438.433831] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:03:07 (1776920587) [ 1441.961129] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:03:10 (1776920590) [ 1445.603910] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:03:14 (1776920594) [ 1448.885659] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:03:17 (1776920597) [ 1452.371932] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:03:21 (1776920601) [ 1455.547115] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1267 sec ========== 01:03:24 (1776920604) [ 1455.673264] LNet: 141618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1455.677200] LNetError: 141618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.686588] LNet: Removed LNI 192.168.203.5@tcp [ 1455.981422] Key type .llcrypt unregistered [ 1455.983364] Key type ._llcrypt unregistered [ 1461.515183] Key type ._llcrypt registered [ 1461.517751] Key type .llcrypt registered [ 1461.658124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1461.667563] alg: No test for adler32 (adler32-zlib) [ 1462.583319] Lustre: Lustre: Build Version: 2.17.51_74_g68f2299 [ 1462.710855] LNet: Added LNI 192.168.203.5@tcp [8/256/0/180] [ 1464.311177] Key type lgssc registered [ 1464.862568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1491.305051] Lustre: Mounted lustre-client [ 1494.154908] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1502.276485] LustreError: 144065:0:(lov_obd.c:786:lov_cleanup()) lustre-clilov-ffff8f0148a34800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1502.298760] Lustre: Unmounted lustre-client [ 1515.204875] Key type lgssc unregistered [ 1515.339331] LNet: 144545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.342265] LNetError: 144545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.353938] LNet: Removed LNI 192.168.203.5@tcp [ 1515.641358] Key type .llcrypt unregistered [ 1515.643503] Key type ._llcrypt unregistered